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ABSTRACT 


This  thesis  describes  a computer  program  for  constructing  a descrip- 
tion of  solid  bodies  from  a set  of  n pictures  of  the  bodies.  The  bodies 
are  assumed  to  be  bounded  by  faces  which  are  quadric  or  planar,  and  they 
are  restricted  to  have  all  their  vertices  formed  by  exactly  three  faces. 

The  pictures  are  taken  from  different  vantage  points,  with  the  restric- 
tion that  a slight  shift  in  vantage  point  will  not  alter  the  topology  of 
the  picture.  It  is  assumed  that  the  program  receives  outline  information 
from  a preprocessor  which  has  extracted  this  information  from  the  pic- 
tures. The  outline  information  (set  of  line  structures)  may  be  imperfect 
in  that  some  Junctions  may  be  erroneously  reported  and  some  lines  may  be 
missing.  However,  all  lines  due  to  shadows  are  assumed  to  have  been 
eliminated  by  the  preprocessor.  ^ 

The  thesis  includes  a technique'for  establishing  the  validity  of  the 
Junctions  presented  by  the  preprocessor  as  well  as  for  matching  corres- 
ponding features  in  the  line  structures  derived  from  the  different  pic- 
tures. New  grammar  rules  for  line-drawing  projections  of  curved  and  planar 

solid  bodies  are  developed.  These  are  useful  in  parsing  the  line  drawings. 

* 

They  have  also  led  to  the  definition  of  a new  family  of  impossible  objects. 
The  program  works  simultaneously  with  all  the  available  line  structures. 

The  parsing  of  every  line  structure  is  supported  dynamically  by  the 
results  gotten  thus  far  from  the  parsing  of  the  other  line  structures. 
Through  the  parsing  of  the  line  structures,  the  use  of  picture  comparison 
and  the  application  of  the  grasmiar  rules,  many  of  the  preprocessor  errors 
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are  detected  and  partly  corrected.  The  program  also  can  provide  feedback 
to  the  preprocessor  in  the  form  of  suggestions  as  to  where  to  look  again 
for  lines  in  the  pictures. 

The  program  utilizes  the  extracted  line  structures  corresponding  to 
the  different  bodies  in  all  the  pictures  to  determine  the  set  of  faces 
(insofar  as  possible)  for  every  body.  Every  face  is  defined  by  an  ordered 
set  of  n-tuples.  The  n-tuples  are  the  matched  lines  and  Junctions  in  the 
n different  pictures.  The  three-dimensional  coordinates  of  the  vertices 
and  the  equations  of  the  faces  can  then  be  determined  from  these  n-tuples. 
The  program  was  written  in  PL/I  and  has  been  tested  on  several  scenes. 


iii 


ACKNOWLEDGMENT 

The  author  wishes  to  express  her  gratitude  to  Professor 
Herbert  Freeman  for  suggesting  the  topic  of  this  research  and  for 
his  guidance  and  encouragement. 

The  author  is  also  grateful  to  Professors  I.  Cederbaum  and 
A.  Ginzburg  of  the  Technion  - Israel  Institute  of  Technology  - for 
their  help. 

This  work  was  supported  by  the  Air  Force  Office  of  Scientific 
Research,  Directorate  of  Mathematical  and  Information  Sciences,  under 
grants  AFOSR  TO-I85U,  AFOSR  75-2755,  and  AFOSR  76-2937.  It  was  also 
partly  supported  by  a grant  from  the  American  Society  for  Technion. 


TABLE  OF  CONTENTS 


Page 

Abstract  * 


Acknowledgment 

iii 

List 

of  Illustrations 

vi 

List 

of  Symbols 

viii 

I. 

Introduction 

1 

1.1 

The  Problem  of  Scene  Analysis 

1 

1.2 

Previous  Work 

3 

1.3 

Basic  Definitions  and  Assumptions 

4 

a)  Definitions  relating  to  the  bodies  in  the  scene 

b)  Preprocessor  and  input  data 

4 

6 

II. 

Grammatic  Properties 

8 

2.1 

Junction  Types 

8 

2.2 

Cyclic  Order  Property 

11 

III. 

Geometric  Considerations 

18 

3.1 

The  Picture  Taking  Process 

18 

3.2 

Transformation  Techniques 

21 

3.3 

Matching  Incomplete  Data  in  Three  Pictures 

22 

3.4 

Junction  Matching 

23 

IV. 

Line 

Matching  and  Data  Recovery 

36 

4.1 

Introduction 

36 

4.2 

Fitting  a Line  to  a Set  of  Points 

37 

4.3 

Line  Matching 

38 

4.4 

Data  Recovery 

46 

4.5 

Example 

52 

4.6 

Matching  Lines  by  Range 

57 

V. 

Final  Description 

60 

5.1 

Region  Assembly  and  Second-Level  Data  Recovery 

60 

5.2 

Object  Formation 

65 

5.3 

Type  of  Face  Determination 

66 

5.»» 

Face  Equations 

67 

VI.  Experimental  Results 

7h 

6.1 

First  Example 

7h 

6.2 

Second  Example 

77 

6.3 

Third  Example 

77 

VII.  Summary  and  Conclusions 

100 

7.1 

Summary 

100 

7.2 

Specific  Contributions 

102 

7.3 

Directions  for  Future  Research 

103 

References 

105 

Index  of  Definitions 


107 


vii 


Figure 


Page 


5.1 

Face  Group  Assembly- 

6l 

5.2 

Three  Configurations  for  Second-Level  Data 
Recovery 

61* 

5.3 

Tree  Search  for  Minimal  Cover  to  Determine 
Curved-Face  Face  Group 

68 

5.1* 

Curved-Face  Determination  (Note  that  Face 
Group  B will  Consist  of  Two  Line  Assemblies 
and  Two  Limbs) 

69 

5-5 

Flow  Chart  for  Determination  of  Type  of  Quadric 
Surface 

70 

6.1 

First  Example  (The  Circled  Junctions  are 
Cyclically  Arranged.  The  Arrow  Point  at 
Data  Recovery  Results.  Broken  Lines  Mean 
"Bnpty  Lines,"  for  which  Only  the  End 
Points  are  Known.) 

75 

6.2 

Second  Scene  Example 

78 

6.3 

Second  Scene:  Picture-1  Input  Data 

79 

6.U 

Second  Scene:  Picture-2  Input  Data 

80 

6.5 

Second  Scene:  Picture-3  Input  Data 

81 

6.6 

Example  2 

82 

6.7 

Computer  Output  Data  for  Example  2 

83 

6.8 

Third  Scene  Example 

89 

6.9 

Third  Scene:  Picture-1  Input  Data 

90 

6.10 

Third  Scene:  Picture-2  Input  Data 

91 

6.11 

Third  Scene:  Picture-3  Input  Data 

92 

6.12 

Example  3 

93 

6.13 

Computer  Output  Data  for  Example  3 

9U 

viii 


LIST  OF  SYMBOLS 


Junction  number  s in  picture  i 
Line  connecting  Junction  A and  B 


The  vector  a 


Center  of  projection  for  picture  plane  i 

C.  . Intersection  between  line  C.C.  and  plane  P 

ijq  i J q 

Distance  from  to 

G^  Distance  from  C^  to  center  of  corresponding  film  plane  frame 

J,  Intersection  between  line  A.  C.  and  plane  P 

lq  i,J  i q 

i*l,2,..n  Picture  plane  index  (for  n pictures) 

i*0  Index  for  points  not  associated  with  any  picture  plane 

J*l,2,..n  Index  for  coordinate  system  associated  with  picture  plane  J 

J=0  Index  for  global  coordinate  system 

Projection  matrix  for  projecting  onto  picture  plane  i 
P^  Picture  plane  i 

T.  Matrix  of  transformation  from  global  system  to  system  of 

1 

picture  plane  i 

X^,Y^,Z^  Axes  of  coordinate  system  associated  with  picture  plane  i 

,Y^  . Coordinates  of  point  k associated  with  picture  plane  i in 

1 yK  1 1 yK 

coordinate  system  J 

0.  Angle  of  rotation  of  X*  axis  from  X°  axis 


Angle  of  rotation  of  Y*  axis  from  X°Y°  plane 
Line  Joining  Junctions  l6  and  12  in  picture  1 


I.  INTRODUCTION 


1.1  The  Problem  of  Scene  Analysis 

When  we  look  at  a good  quality  picture  of  a scene  containing  a 
number  of  three-dimensional  objects,  we  are  usually  able  to  "understand" 
the  scene;  that  is,  we  are  able  to  perceive  the  real  physical  nature  of 
the  objects.  The  reason  for  this  is  that  we  have  seen  similar  scenes 
before  and  at  those  times  were  able,  by  a combination  of  touch  and  addi- 
tional views  of  the  scene,  to  develop  the  ability  of  relating  pictures 
of  three-dimensional  objects  to  the  objects  themselves.  The  same 
applies  — perhaps  with  the  need  for  slightly  more  specialized  learning  — 
to  the  understanding  of  scenes  depicted  in  terms  of  line  drawings,  such 
as  a draftsman  might  prepare.  A picture  (or  line  drawing)  of  a scene  is 
merely  a sentence  in  a language  that  we  have  learned.  As  with  other 
languages,  difficulties  with  understanding  arise  if  the  sentence  con- 
tains words  whose  meaning  is  not  known  to  us  or  if  it  contains  unfamiliar 
syntactical  structures.  Further,  sudden  severe  changes  in  context  will 
disorient  us  because  "picture  languages,"  like  other  natural  languages, 
are  strongly  context-dependent. 

A scene  may  be  incomprehensible  from  a picture  because  the  picture 
contains  errors.  For  example  a line  drawing  (which  is  a specialized  type 
of  picture)  may  have  some  line  segments  missing.  In  such  a case  under- 
standing of  the  scene  can  sometimes  be  gained  by  using  a set  of  pictures 
(or  set  of  line  drawings)  describing  the  scene  from  different  vantage 
points.  Although  the  pictures  may  each  be  defective  in  some  way,  collec- 
tively they  may  make  possible  the  unique  interpretation  of  the  scene. 
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In  computer  scene  analysis  an  image  of  a three-dimensional  scene  is 
converted  into  an  array  of  pixels,  with  each  assigned  a number  corres- 
ponding to  the  average  grey  value  in  that  local  area  of  the  image.  The 
image  is  then  preprocessed  to  extract  features  suitable  for  scene  inter- 
pretation. One  common  scheme  is  to  extract  the  lines  that  are  believed 
to  correspond  to  the  edges  of  the  objects.  Because  of  glare,  haze, 
shadows,  texture  and  other  lighting  conditions,  some  edges  may  be 
partially  or  wholly  obscured  and,  for  basically  similar  reasons,  some 
spurious  lines  may  be  mistakenly  thought  to  correspond  to  edges.  The 
scene  analysis  program  must  extract  as  much  useful  information  as  possible 
from  each  picture  and  then  relate  this  information  to  similar  informa- 
tion devised  from  other  pictures  of  the  same  scene  (i.e.,  from  pictures 
taken  from  different  vantage  points  or  under  different  lighting  condi- 
tions). The  use  of  multiple  pictures  is  useful  both  for  obtaining 
information  about  the  three-dimensional  nature  of  the  objects  as  well  as 
for  resolving  ambiguities  due  to  imperfectly  extracted  edge  data. 

We  can  describe  complex  bodies  in  terms  of  volumes,  that  is,  as 
assemblies  of  simple  bodies,  or  we  can  describe  them  in  terms  of  their 
surfaces,  using  faces,  edges  and  vertices.  The  second  type  of  descrip- 
tion is  particularly  effective  for  polyhedra.  But  it  is  also  suitable 
for  curved  bodies,  especially  man-made  bodies,  for  which  every  distinct 
face  can  be  described  by  a relatively  simple  mathematical  equation. 

This  thesis  is  restricted  to  the  analysis  of  scenes  containing  only 
bodies  with  planar  or  quadric  faces;  however,  the  results  could  be  easily 
modified  to  deal  also  with  bodies  whose  curved  faces  are  close  to  quadric 
surfaces,  and  this  includes  a big  subset  of  man-made  bodies. 
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1.2  Previous  Work 

The  description  scheme  based  on  the  use  of  volumes  is  exemplified 
by  Agin  (l)  who  described  curved  bodies  as  combinations  of  generalized 
cylinders,  each  described  by  its  axis  and  cross  section.  Schemes  using 
the  faces-edges-vertices  approach  are  almost  entirely  limited  to  poly- 
hedra;  they  include  Roberts  (ll ) , Guzman  (7),  Falk  (6),  Huffman  (8), 

Clowes  (4),  Waltz  (18 ) , and  many  more.  Nearly  all  of  them  utilized  only 
a single  picture  of  the  scene.  Although  all  were  aware  of  the  pre- 
processor limitations  in  extracting  complete  and  valid  edge  data,  the 
approach  varied.  Some  maintained  that  a program  analyzing  an  (unreal- 
istic) perfect  line  drawing  also  contributes  to  understanding  the 
process  of  perception,  and  indeed,  their  work  provided  much  insight  and 
guidance  for  computer  scene  analysis.  They  include  Guzman  (7)  with  his 
classification  of  Junctions,  Huffman  (8)  and  his  elegantly  expressed 
edge  labelling  scheme,  Clowes  (4)  with  his  special  notation  and 
Waltz  (18 ) , who  made  extensive  use  of  grammatic  rules  for  analyzing 
perfect  line  drawings  with  shadows.  There  is  also  the  other  group, 
exemplified  by  Falk  (6),  who  assumed  that  the  input  data  was  imperfectly 
extracted  from  real  pictures  and  imposed  severe  restrictions  on  the  scene 
(such  as  preknowledge  of  the  set  of  bodies).  It  is  this  writer's  opinion 
that  the  restriction  to  data  from  a single  picture  by  all  these  researchers 
created  unnecessary  difficulties  and  caused  much  effort  to  be  expended  on 
finding  solutions  to  the  wrong  problems. 

The  extension  of  the  work  from  polyhedra  to  curved  bodies,  even  if 
only  to  a restricted  family  of  curved  bodies,  increases  the  difficulties 
to  an  extent  which  may  not  be  apparent  at  first  glance.  Some  of  these 
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difficulties  are  pointed  out  in  this  thesis.  A partial  extension  to 
curved  bodies  can  be  found  in  the  work  by  Chien  and  Chang  (3).  They 
dealt  with  bodies  whose  faces  are  planar,  conic  or  cylindric,  in  which 
every  curved  face  must  be  bounded  by  planar  faces.  They  assumed  a per- 
fect line  drawing  for  their  input  data.  Their  thesis  can  be  considered 
an  extension  of  Guzman's  work. 

As  already  mentioned,  virtually  all  of  the  early  work  utilized  data 
from  only  one  picture  of  the  scene.  The  use  of  multiple  pictures  taken 
from  different  vantage  points  was  described  in  the  recently  published 

paper  by  Underwood  and  Coates  (19).  Their  program  analyzes  convex  poly- 

« 

hedra  photographed  from  different  vantage  points,  a single  polyhedron  in 
every  set,  where  the  input  data  is  assumed  to  be  perfect.  The  set  of 
multiple  pictures  is  thus  not  used  to  verify  unreliable  data  but  merely 
"to  have  a look"  on  all  sides  of  the  body. 

In  this  thesis  we  shall  show  how  a program  can  be  made  to  "understand" 
a set  of  bodies  with  planar  or  quadric  faces,  utilizing  defective  data 
extracted  from  a set  of  multiple  photographs.  No  preknowledge  of  the 
bodies  is  assumed,  but  certain  general  properties  of  the  scene  - such  as 
the  existence  of  precisely  three  faces  in  every  vertex  - are  assumed. 

1.3  Basic  Definitions  and  Assumptions 

a)  Definitions  relating  to  the  bodies  in  the  scene.  Many  of  the  follow- 
ing terms  have  been  adapted  from  those  used  by  Woon  ( 20 ) and  Clowes  ( U ) . 
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A quadric  surface  is  the  locus  of  points  obeying  the  equation 

Q(x,y,z)=uTAu  -0  (l) 

T 

where  u*(x,y,z,l)  is  a vector  and  A is  the  coefficients  matrix 

i ai i *5  ai2  h ais  h a,7| 


A surface  is  either  a quadric  surface  or  a plane. 

An  edge  is  all  or  part  of  the  intersection  of  two  surfaces;  the 
intersection  is  bounded  by  one  or  two  other  surfaces  or  is  closed  on 
itself. 

A vertex  is  the  intersection  of  three  or  more  edges. 

A face  is  a portion  of  a surface  bounded  by  edges  or  closed  on 
itself.  A boundary  is  a closed  chain  of  edges  bounding  a face.  A single 
face  may  have  several  boundaries. 

A body  is  a closed,  connected  part  of  the  3D  space,  delimited  by  a 
finite  number  of  faces. 

A scene  is  a set  of  bodies. 

A projection  is  a central  projection  of  a 3D  entity  onto  the  picture 
plane. 

A limb  is  the  locus  of  the  points  on  a quadric  surface  that  are 
tangent  to  the  projecting  rays  and  do  not  lie  on  an  edge. 

A virtual  vertex  is  the  intersection  of  an  edge  and  a limb. 
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A region  is  a connected,  visible  part  of  the  projection  of  a face. 
The  projection  of  a face  may  result  in  zero,  one,  or  more  regions. 

A line  is  the  projection  of  an  edge  or  a limb.  The  line  projected 
by  a limb  will  also  be  called  a limb  when  this  will  not  cause  any  confu- 
sion. A line  may  be  straight  or  curved. 

A Junction  is  the  intersection  of  two  or  more  lines,  and  is  thus 
either  the  projection  of  a vertex  or  of  a virtual  vertex,  or  is  a result 
of  obscuring  part  of  an  edge  or  limb  by  a face. 

An  object  is  the  set  of  regions,  lines,  and  Junctions  corresponding 
to  a single  body. 

We  impose  the  following  restrictions: 

1.  Every  vertex  in  the  scene  is  formed  by  exactly  three  surfaces, 
and  belongs  to  exactly  three  edges. 

2.  Smooth  transition  between  two  different  faces  is  not  allowed 
(i.e.,  the  derivative  must  be  discontinuous  across  the  edge). 

3*  The  camera  position  is  assumed  to  be  "general."  For  example, 
the  projections  of  different  vertices  may  not  coincide  in  any 
picture.  If  a vertex  does  not  belong  to  an  edge,  the  projection 
of  the  vertex  may  not  coincide  with  the  projection  of  the  edge 
in  any  picture. 

1».  No  limb  passes  through  a vertex, 
b)  Preprocessor  and  input  data.  It  is  assumed  that  there  is  a pre- 
processor which  extracts  the  lines  and  Junctions  from  the  picture 
(photograph)  of  the  scene.  The  extraction  of  curved  lines  has  been  the 
subject  of  work  by  Ramer  ( 10)  and  Agin  (l).  There  is  also  the  work  of 
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Shirai  and  Tsuji  (15 )»  who  extracted  straight  lines  from  photographs 
of  polyhedra,  getting  rid  of  shadows  at  the  expense  of  missing  some 
visible  lines.  Our  preprocessor  is  assumed  to  be  a combination  of 
Shirai  and  Tsuji's  (ORing  and  ANDing  of  a sequence  of  pictures  taken 
from  the  same  vantage  point  under  different  conditions  of  illumination 
to  eliminate  shadows)  and  of  Ramer's  (curved-line  extraction  and  deter- 
mination of  Junctions).  The  output  of  such  a preprocessor  is  the 
input  to  the  scene  description  program  described  here.  The  assumptions 
about  the  input  data  thus  are  as  follows: 

t 

1.  There  are  no  extraneous  lines. 

2.  Seme  valid  lines  (or  parts  of  them)  may  be  missing. 

Since  the  kind  of  preprocessor  assumed  here  was  not  in  fact  avail- 
able, the  data  for  testing  of  the  scene  description  program  had  to  be 
extracted  visually  from  sets  of  actual  photographs. 


II.  GRAMMATIC  PROPERTIES 


2.1  Junction  Types 

In  accordance  with  the  assumptions  stated  in  the  previous  section, 
junctions  can  have  no  more  than  three  lines.  They  can  thus  be  classi- 
fied into  the  following  types: 


Type 


■ \K 


Type  T 


Type  V 


V" 


Type  A 


Types  W,  V,  Y and  T are  the  well-known  types,  generalized  here  (7). 
Types  Y and  W must  be  projections  of  vertices.  Type  T results  from  the 
covering  of  part  of  an  edge  or  limb  by  a face.  Types  A and  S are  the 
projections  of  virtual  vertices,  where  the  line  e is  the  projection  of 
a limb.  Characteristic  to  Junction  types  A and  S is  that  all  the  lines 
in  the  Junction  have  a common  tangent.  A type-V  Junction  is  either  the 
projection  of  a vertex  whose  third  line  is  not  visible  or  it  is  a Junc- 
tion which  could  be  of  either  type  Y or  W but  has  one  line  missing,  or 
of  type  T with  part  of  the  bar  missing.  If  an  S junction  should  be 
missing  a line,  it  can  look  like  a type-A  Junction  (f"“  , like  a 

type-V  Junction  , or  be  undetectable  . (The  dots 

indicate  the  missing  line.)  An  illustration  of  the  different  junction 
types  is  given  in  Fig.  2.1. 

The  detection  of  a type-A  Junction  is  difficult  since  this  type  of 
Junction  is  characterized  by  a continuous  transition  between  two  lines. 
The  two  lines  are  governed  by  different  equations  and  on 'the  basis  of 
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this  segmentation  of  the  continuous  contour  into  the  two  line  segments 
should  be  possible.  Such  segmentation  was  accomplished  by  Agin  (l)  and 
Albano  (2). 

Through  Junctions  of  types  S and  A,  a limb  can  be  identified.  A 
limb's  importance  lies  in  that  it  is  projection-dependent  and  has  no  match 
in  a picture  taken  from  a different  vantage  point.  Also,  it  conveys 
information  regarding  the  type  of  the  face  (cylinder,  cone,  ellipsoid, 
hyperboloid,  etc.)  and  is  useful  in  finding  the  face's  equation.  The 
type  identification  of  each  Junction  is  assumed  to  be  supplied  by  the 
preprocessor. 

It  would  be  useful  to  have  an  extended  (or  generalized)  Junction 
labelling  scheme  based  on  Huffman's  polyhedral  labelling  ("-"  for  a line 
projected  by  a concave  edge,  "+"  for  a convex  edge,  and  a directed  arrow 
for  a convex  edge  created  by  a front  and  a back  face*).  However,  for 
curved  objects  the  labelling  of  a line  may  not  always  be  consistent  and, 
therefore,  the  usefulness  of  Junction  labelling  here  is  of  doubtful  value 
since  it  cannot  be  propagated  along  the  line.  (See  for  example  the 
labelled  line  in  Fig.  2.1).  The  distinction  in  this  thesis  between  a Y 
and  a W Junction  is  in  keeping  with  past  practice  rather  than  for  any 
practical  reason. 


* A front  face  is  a face  for  which  the  outward  normal  (at  least  at  the 
edge  in  question)  has  a component  directed  toward  the  vantage  point. 
For  a back  face,  the  component  is  directed  away  from  the  vantage 
point.  See  Woon  and  Freeman  (21). 
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2.2  Cyclic-Order  Property 

Let  us  define  the  depth  Index  of  a line  as  the  number  of  faces 
modulo-2  between  the  edge  corresponding  to  this  line  and  the  center  of 
projection.  As  mentioned  before,  every  vertex  belongs  to  exactly  three 
edges.  We  define  for  these  edges  a cyclic  order  in  the  vertex,  as  the 
order  induced  by  "walking  around"  the  vertex  in  a clockwise  manner,  and 
numbering  the  edges  in  the  order  1,  2,  3.  The  cyclic  order  will  retain 
its  clockwise  sense  in  any  projection,  irrespective  of  the  shape  of  the 
vertex,  under  either  of  the  following  two  conditions: 


1.  All  three  lines  have  an  even  depth  index. 

2.  One  of  the  three  lines  has  an  even  depth  index  and  two  have 
an  odd  one. 

The  cyclic  order  will  be  reversed  (become  counter-clockwise)  in  a pro- 
jection if: 

1.  One  of  the  lines  has  an  odd  depth  index  and  two  have  an  even 
one. 

or 

2.  All  three  lines  have  an  odd  depth  index. 

The  different  cases  are  illustrated  by  the  example  shown  in  Fig.  2.2. 

If  we  trace  out  the  edges  of  a boundary,  we  consistently  change 
edges  at  the  vertices,  either  always  in  a decreasing  cyclic  order  or 
always  in  an  increasing  cyclic  order.  Let  us  always  choose  to  trace 
edges  in  an  increasing  order.  Then  we  shall  traverse  every  edge  in  two 
different  directions  as  we  walk  around  the  two  faces  that  share  it. 

(See  the  two  arrowed  lines  along  the  line  AB  in  Fig.  2.2). 


I 
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From  the  foregoing  we  conclude  that  when  a vertex's  projection  has 
three  lines,  their  cyclic  order  is  determined  by  their  arrangement  in  the 
picture.  However,  when  one  line  is  missing,  the  cvr"lic  order  not 
known.  Let  us  define  the  relation  AB  < AC  to  mean  that  in  Junction  A 
the  line  AC  follows  the  line  AB  immediately  in  the  cyclic  order.  Then  if 
A is  a two-line  Junction,  we  have  either  the  relation  AB  < AC  or  the 
relation  AC  < AB. 

Knowledge  of  the  cyclic  order  at  a junction  is  crucial  to  the 
assembly  of  the  lines  belonging  to  a single  region.  Therefore,  it  is 
important  to  have  a means  for  determining  the  cyclic  order  in  a Junction 
when  the  order  is  not  given  naturally.  Some  rules  and  techniques  are 
stated  in  this  section,  with  additional  rules  given  later  in  the  thesis. 

We  define  a line  assembly  (LA)  as  the  directed  path  followed  in 
tracing  out  the  lines  corresponding  to  a single  boundary  - or  any 
continuous  part  of  it  - in  increasing  cyclic  order.  We  denote  an  LA  in 


picture  i by  the  ordered  set  of  Junctions  A.  , A.  visited  in 

1,1  i,n 


the  course  of  the  trace.  Now  it  follows  from  the  definition  that  for 

every  1 < k < n we  have  A.  , A.  ,,  .<  A.  . A.  . , ..  Two  LA's  are  said 

i,k  i , ( k-1 ) i,k  i,(k+l) 

to  be  distinct  if  they  trace  out  lines  corresponding  to  two  distinct 

boundaries . If  for  two  LA's,  A.  , ...  A.  and  A,  , ...  A, 

i ,1  i,n  j,l  Jrfi 

1.  A,  “A,  , and  there  is  at  least  one  A.  different  from 

1*1  I *n  J 


or 


every  A^ 


2.  There  are  two  successive  Junctions  in  the  two  LA's  such  that 
“d  Ai,(k-1)  “ AJ,(£+1) 


Ai*k*AJ,t 


- Ik  - 


or 


3.  A.  =A  and  the  relation  A A < A.  , .A.  exists. 

i,n  J,1  J,1  J ,2  i,(n-l)  i,n 

Then  the  two  LA's  are  distinct. 

Two  rules  forcing  a cyclic  order  in  a junction  can  be  stated  now: 

» 

Rule  1:  If  we  have  an  LA,  A.  .....  A.  in  which  A.  . =A. 

i,l  i,n,  i,l  i,n 

then  we  must  have  in  A.  , the  relation  A.  „ A.  , . . < A.  . A . . . 

i,l  i ,1  i , ( n-1 ) i ,1  i,2 

For  example,  in  Fig.  2.3(a)  we  have  the  LA  B,C,D,A,B,  (The  LA  is  shown  as 
an  arrowed  line).  The  lines  in  B are  thus  forced  to  have  the  relation 
BA  < BC 

Rule  2:  If  we  have  two  distinct  LA's.  A A.  and 

i ,1  i,n 

A,  .....  A,  , such  that  A =A.  , and  A,  , , >?A.  0,  then 

j,l  J ,m  j ,m  i,l  j, (m-1)  1,2 

we  must  have  in  A.  , the  relation  A.  , A.  . < A,  A . 

i ,1  i,l  i ,2  j,m  j ,(m-l) 

For  example  in  Fig.  2.3(b)  we  have  the  two  distinct  LA's:  E,F,C,B  and 

B,A,G,E.  They  are  distinct  because  of  the  cyclic  order  already  established 
in  Junction  B by  Rule  1.  Now  by  Rule  2 the  relation  EF  < EG  must  hold  in 
E.  (This  in  turn  establishes  the  LA  M,F,E,G,N,0) . . 

A more  complicated  strategy  may  be  adopted  to  force  the  correct  order 
at  a Junction  when  these  two  rules  cannot  be  applied  directly.  One  assumes 
a cyclic  order  in  the  Junction  and  tries  to  reach  a contradiction  by  a 
sequence  of  inductive  steps.  If  a contradiction  is  reached,  the  cyclic 
order  opposite  to  the  one  selected  is  forced  on  the  junction.  For  exairple, 
in  Fig.  2.3(c)  we  assume  an  order  in  Junction  R such  that  RQ  < RN.  Then 
we  can  establish  the  LA’s  P,Q,R,N,G,A,D  and  D,C,F,M,P.  They  are  distinct 
as  implied  by  the  natural  order  in  D,  and  hence  by  Rule  2 force  in  P the 
relation  PQ  < PM.  At  the  same  time  we  have  the  two  LA's  R,Q,0  and  0,N,R 


'ossa gsnr-' 
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which  are  distinct  due  to  the  assumed  order  in  R.  By  Rule  2,  the 
relation  ON  < OQ  is  forced  in  0.  This  in  turn  establishes  the  LA 

M,F,E,G,N,0,Q,P,M.  Now  by  Rule  1 we  must  have  in  M the  relation  MP  < MF, 

♦ 

which  contradicts  the  natural  order  in  M.  This  sequence  of  deductions 
was  based  only  on  the  assumption  that  the  relation  RQ  < RN  exists  at  R, 
which  lets  us  conclude  that  the  relation  RN  < RQ  must  exist  at  R.  This 
establishes  the  LA  0,N,R,Q,0,  and  by  Rule  1 the  relation  OQ  < ON  is 
induced  at  0.  We  have  thus  been  able  to  find  the  cyclic  order  at  U of  the 
5 original  unordered  Junctions.  Junction  P remains  undetermined.  The 
final  ordering  of  the  Junctions  is  given  in  Fig.  2.3(d). 

A byproduct  of  the  foregoing  is  a new  class  of  impossible  objects, 
in  addition  to  those  described  by  Hufftaan  (8).  The  objects  can  be  identi- 
fied as  being  impossible  if  their  structure  is  such  as  to  lead  to  contra- 
dictory cyclic  order  at  their  Junctions.  In  Fig.  2.k  we  have  some 
instances  of  impossible  objects.  In  Fig.  2.U(a)  the  LA  A,E,D,C,B,A  forces 
in  A - by  Rule  1 - the  relation  AB  < AE  contradicting  the  natural  order 
in  A.  In  Fig.  2. Mb)  the  relation  AD  < AB  is  forced  in  A,  again  con- 
tradicting the  natural  order  at  A.  In  Fig.  2. Me)  the  LA  A,E,F,B,A  forces 
the  relation  AB  < AE  in  A,  whereas  the  LA  A,B,C,D,E,A  forces  the  con- 
tradicting relation  AE  < AB.  The  same  occurs  in  Fig.  2.U(d),  where  the 
relation  AB  < AE  is  forced  at  A by  the  LA  A,E,F,B,A  and  the  contra- 
dictory relation  AE  < AB  is  forced  by  the  LA  A,B,C,D,E,A.  Note  that 
the  two  polyhedra  (a)  and  (d)  pass  Huffman's  labelling  test. 


III.  GEOMETRIC  CONSIDERATIONS 


3. 1 The  Picture  Taking  Process 

Th»  basic  input  data  consists  of  a set  of  three  pictures  taken  from 

three  different  vantage  points.  A coordinate  system  Xi,Y^,Zi  is  defined 

for  each  picture  plane  and  related  to  a global  coordinate  system 
0 0 0 

X ,Y  ,Z  in  which  the  final  description  of  the  scene  is  determined.  Each 
picture's  coordinate  system  has  its  origin  (X^  ,Y?  ,Z?  ) on  the 
corresponding  camera  axis.  Its  Y*  axis  coincides  with  the  camera  axis 
and  makes  an  angle  with  the  X^Y^  plane.  Its  X*  axis  lies  in  a plane 
parallel  to  the  plane  X^Y°  and  is  rotated  by  an  angle  0^  from  the  X°  axis 
(Fig.  3.1).  Superscripts  are  used  to  denote  the  particular  coordinate 
system,  with  superscript  0 denoting  the  global  system.  First-position 
subscripts  refer  to  the  picture  plane  and  second-position  subscripts 
identify  particular  points. 

To  relate  a picture  coordinate  system  to  the  global  coordinate  system 
one  must  know  the  angles  of  rotation  6^  and  <(i^  and  the  amount  of  trans- 
lation X^  o’^i  O'  T°  obtain  the  anSles  of  rotation  one  measures, 

for  every  vantage  point,  two  points  on  the  camera  axis.  One  point  is  the 

center  (X?  m,yP  „,Z?  „)  of  the  film  plane  frame  in  the  camera,  and  the 
1 ,N  1 ,N  1 ,N 

other  is  any  convenient  point.  The  cosine  and  sine  of  the  angles  can  now 
be  calculated.  The  distance  between  the  center  of  projection  and  the 
point  (X?  „,Y®  „,Z?  „)  is  denoted  by  G, . The  distance  between  the  center 

X )N  1 y I»  i 9U  X 

of  projection  and  the  point  (X°  Q,Y°  ^,Z^  Q)  is  denoted  by  F^  The  ratio 
■ G^/F^  is  determined  by  measuring  corresponding  diagonals  of  both 
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In  order  to  compute  for  every  picture  we  look  at  a set  of  pre- 

measured points  {(X°  ,Y°  ,z|?  ,)},  J=1  ...  n,  whose  set  of  projections 

o»j 

{ (X^  Z * ) } 

i,j’  i.J  J*1  •••  n,  i*l  ...  3,  are  measured  in  every  picture.  To 
minimize  the  sum-of-squarea  differences  between  the  calculated  projec- 
tions and  the  measured  projections  we  require  that 

( l (xj  - x}  )2  ♦ l (zj  - 2}  )2)  • 0,  i*l. . .3  (5) 

for  every  picture.  The  tnree  equations  can  now  be  solved  for  F . 


3.2  Transformation  Techniques 

Interrelation  between  the  three  picture  coordinate  systems  can  be 
achieved  by  transforming  the  coordinates  via  the  global  system.  The 
translation  (X^  0,Yi  0,Zi  0^  can  be  exPreS3ed  as  follows: 


“ — 

— — 

— 

x° 

1,0 

X°  „ 
i,N 

-cosff^  sin0i 

Y?,o 

s 

Y?.» 

+ FJ(Ri^l) 

COS^  COS0^ 

*0_ 

. <». 

sirxt^ 

(6) 


The  matrix  of  transformation  from  the  global  system  to  the  system 
of  pictures  i is: 


T.  ■ 


cos8 . 


sin0. 


i i 

-cos^  sin0i  cos^  co80j  sin^j 


-sim^  cos 0^  -sin^  cos0^  cos$^ 


-<o  cosV  *1,0  slnei) 
-<-Xl,0  cos4,i  sinV  YJ,0 

cos<J>^  cos0^+  q sin<}>^) 

•(ltl,0  »‘"*i  »!»«!  - <0 

^in^  cos01  + Q cos<^i ) 


(7) 


0 


0 


1 
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Thus  if  Xq  j Yq  j»Zq  j are  the  coordinates  of  a point  expressed  in 

the  coordinate  system  of  picture  J and  X*  Y*  T,Z*  _ are  the  coordinates 

U,J  0,J 

of  the  same  point  in  the  coordinate  system  of  picture  i,  we  have 


X* 

XJ 

0,J 

0,J 

* T T’1 

*1  -J 

YJ 

0,J 

ZJ 

Z0,J 

1 

1 

— 

Other  parameters  of  importance  are  the  coordinates  (X°  ,Y°  ,Z°  ) of 

1 jw  1 f v 1 yL 

the  center  of  projection  (center  of  lens)  for  every  picture: 


n 

~ n n 

r — 

YU 

i,C 

X° 

i,N 

-cos^  sinS^ 

Y° 

i.C 

S 

Yi,N 

+ Fi  Ri 

cos<(>^  cos0^ 

sin^ 

3.3  Matching  Incomplete  Data  in  Three  Pictures 

The  line  structure  data  extracted  from  the  pictures  may  he  incom- 
plete because  of  occlusion  or  because  of  preprocessor  failure.  Let  us 
consider  the  data  imperfections  due  to  preprocessor  failure.  They  can 
be  divided  into  two  groups: 

(l)  Data  Imperfections  involving  .junctions. 

a.  Missing  Junctions. 

b.  Palse  Junctions  - for  example,  when  half  of  the  bar  of  a 

T Junction  is  missing,  causing  it  to  look  like  a V Junction. 


•-  --  - - 
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c.  A V Junction  that  is  actually  a Y or  W Junction  with  one  line 
missing. 

Note  that  the  V in  subgroup  c,  although  of  wrong  type,  is  the  projection 
of  a vertex;  this  is  not  true  for  V in  b. 

(2)  Data  imperfections  involving  lines. 

a.  Part  of  a line  is  missing  and  the  broken  end  is  open  (dangling). 

b.  Part  of  a line  is  missing  and  a Junction  of  wrong  type  is 
created  at  the  broken  end. 

c.  The  whole  line  is  missing. 

We  call  a Junction  valid  when  it  is  a projection  of  a vertex. 

Thus,  Junctions  of  types  A,  S,  and  T are  not  valid,  junctions  of  type 
Y and  W are  valid  and  for  V Junctions  we  do  not  know.  The  precise  nature 
of  a V Junction  must  be  left  undecided  until  more  evidence  is  collected, 
as  will  be  explained  later. 

A Junction  of  type  S which  is  missing  a line  and  is  reported  as  an 
A Junction,  does  not  cause  any  confusion  and,  therefore,  is  not  counted 
as  an  imperfection. 

3-1*  Junction  Matching 

In  order  to  arrive  at  a true  description  of  the  3D  scene  in  spite  of 
some  data  imperfections,  we  compare  the  line  structures  from  the  three  pic- 
tures against  each  other  and  use  the  information  found  in  one  picture  to 
check  the  information  found  in  another. 

Let  us  start  with  Junctions.  We  have  two  goals.  One  is  to  determine 
which  of  the  V junctions  are  valid,  or,  to  put  it  differently  for  which  V 
Junction  Is  there  enough  evidence  to  support  such  an  assumption.  The 
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second  is  to  group,  from  the  different  pictures,  Junctions  that  are  the 
projection  of  the  same  vertex.  From  here  on  in  this  section  whenever 
we  use  the  term  "Junction"  we  shall  mean  a V,  Y or  W Junction. 

If  two  Junctions,  one  each  in  two  different  pictures,  are  projec- 
tions of  the  same  vertex,  they  must  obey  certain  geometric  rules.  Let 

P , C and  A be  the  picture  plane,  the  center  of  projection,  and 
1 1 1 , J 

the  projection  of  vertex  J,  respectively,  for  picture  i.  Then  for  two 

pictures,  i and  J,  C. , C.,  J,  A.  T,  and  A.  . are  coplanar,  and  the  lines 

1 J J 

C.A  ,C  A intersect  any  plane  P in  three  colinear  points: 

i J J>“  <1 

C4  , , J.  and  J , respectively  (Fig.  3.2).  We  shall  call  the  line 
ljq  1(1  jq 

formed  by  these  points  a match  line.  Thus  if  we  re-project  into  the  plane  P 

Q 

Junction  A.  _ from  C , , Junction  A.  T from  C4,  and  C.  from  C.,  we  can  check 

1 * J ***  J 1 J 

whether  this  condition  is  satisfied.  Two  Junctions  from  different  pictures 

that  obey  this  condition  are  said  to  be  matchable.  Obviously,  if  a vertex 

I is  in  the  plane  C, , C , J then  A.  _ and  A are  matchable  and  so  are 
1 J J *” 

A.  T and  A . To  resolve  such  ambiguities,  we  refer  to  the  third  picture, 

1 *«  J * I 

picture  k.  Now  we  require  that  be  not  colinear.  Then  if 

Ai  J’  AJ  J’  an<i  Ak  J are  the  tJiree  Pr°Jections  of  vertex  J,  the  three 

different  match  lines  on  plane  P form  a triangle  whose  vertices  are  J . 

Q iq’ 

and  J^.  (Fig.  3.3).  We  call  the  set  of  three  Junctions,  from  differ- 
ent pictures,  that  are  matchable  in  pairs,  a triple.  By  forming  triples 
we  can  eliminate  many  pairs  that  are  matchable  but  are  not  projections  of 
the  same  vertex.  However,  even  in  the  pure  geometric  case  we  may  have 
three  projections  of  three  different  vertices  A,B,  and  D that  form  a triple. 

(rig.  3.*»). 


1 
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Fig.  3.3  The  Match-Line  Triangle  for  Vertex  J 
(Match  Lines  Shorn  Bold) 
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Fig.  3.1*  Triple  Ambiguity  (A,  ,B.  ,D,  ) 

kq  iq  Jq 
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We  are  dealing  here  with  real  data;  thus  we  must  accept  A.  and 

1 , J 

A as  matc'nable  even  if  J.  is  distant  from  the  match  line  C.,  J 

iq  ljq  jq 

by  an  amount  less  than  some  threshold.  The  threshold  distance  should 
be  a function  of  the  precision  of  the  measurements.  In  the  approach 
taken  here  the  plane  P was  taken  to  coincide  with  one  of  the  picture 

q 

planes,  and  the  threshold  was  taken  as  the  lesser  of  (l)  half  the  dis- 
tance of  the  two  closest  Junctions  and  (2)  some  experimentally  deter- 
mined fraction  of  the  picture  diagonal. 

Relaxing  the  colinearity  requirement  on  the  matching  lines  increases 
the  ambiguity  of  matching.  The  larger  the  threshold  distance,  the 
greater  the  number  of  ambiguous  cases.  This  also  affects  the  forming  of 
triples.  The  flatter  the  match-line  triangle  becomes,  the  more  acute 
will  be  the  angles  between  pairs  of  match  lines.  As  can  be  seen  in  Fig. 
3.5,  the  more  acute  these  angles,  the  greater  the  chance  that  a point 
will  be  caught  within  the  threshold  distance  of  two  wrong  match  lines 

(e.g.,  I within  match  lines  B B.  and  B.  B,  ).  We  can  thus  expect  to 
iq  Jq-iq  iq  kq 

obtain  triples  where  two  Junctions  are  projections  of  the  same  vertex 
but  the  third  is  not. 

Two  matchable  Junctions  that  are  projections  of  the  same  vertex  are 
said  to  match  each  other.  A triple  in  which  the  Junctions  match  each 
other  is  called  a match  triple. 

To  establish  matches,  we  find  for  every  Junction  all  matchable 
Junctions  in  the  two  other  pictures.  Then  we  form  all  the  triples.  To 
find  the  match  triples  in  the  set  of  all  triples,  we  have  to  use  some 
picture  context.  The  more  severe  the  conditions  causing  ambiguity,  the 
more  the  need  for  context  information. 
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The  kind  of  context  we  shall  use  is  line  connections  between  Junc- 
tions. If  we  have  three  lines  - each  in  a different  picture  - whose 
ends  form  two  triples,  the  two  triples  are  each  considered  to  be  a match 
triple.  (The  chance  of  naming  a wrong  triple  as  a match  triple  still 
exists  but  is  now  reduced  greatly). 

A vertex's  projection  may  be  missing  in  one  or  more  pictures,  as 
already  mentioned.  It  is  desirable,  therefore,  to  establish  matches 
between  matchable  pairs.  Naturally,  more  context  must  be  used  to 
establish  a match  on  the  basis  of  only  a two-picture  comparison.  The 
pair  match  is  done  in  two  passes  and  only  after  the  triple  match  phase 
is  completed.  (it  reduces  the  number  of  free  candidates  and,  therefore, 
the  chances  of  error).  We  select  a set  of  three  junctions,  one  of  them  - 
call  it  the  middle  - connected  by  lines  to  the  other  two,  that  are 
matchable  to  a set  of  three  junctions  connected,  in  the  same  way  in 
another  picture.  In  the  first  pass  we  require  that  the  middle  junction 
will  be  of  type  Y or  W and  that  the  two  lines  have  the  same  cyclic  pre- 
cedence in  the  two  Junctions.  The  three  junctions  are  then  assumed  to 
match  the  three  Junctions  in  the  other  picture.  In  the  second  pass  we 
allow  one  or  both  of  the  middle  Junctions  to  be  of  type  V and  do  not 
check  the  cyclic  precedence  (it  is  not  known  yet  for  the  two-line 
Junction).  The  second  pass  is  a little  more  hazardous,  not  having  the 
ability  to  use  cyclic  precedence,  but  since  it  is  carried  out  after 
most  Junctions  are  already  matched,  there  is  only  a small  chance  of 
making  an  error.  We  also  accept  two  Junctions  forming  a loop,  which  are 
matchable  to  two  other  Junctions  forming  a loop,  as  matching  to  the  other 


two. 
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If  one  Junction  in  a triple  is  matched  to  the  other  two  by  two 
applications  of  a pair-wise  match,  we  consider  the  triple  to  be  a 
match  triple;  that  is,  we  consider  the  remaining  two  as  matching  also. 

Every  V Junction  that  is  matched  to  a Junction  in  another  picture 
is  marked  as  valid  (although  it  may  miss  a visible  line).  The  match 
supplies  additional  evidence  that  the  Junction  is  a projection  of  a 
vertex. 

An  illustrative  example  is  shown  in  Fig's.  3.6  to  3.8.  The 
figures  consist  of  three  pictures  of  the  same  scene,  taken  from  differ- 
ent vantage  points.  In  every  picture  seme  lines,  or  parts  of  lines  are 
missing.  Many  Y and  W Junctions  are  thus  reported  as  V Junctions,  but 
the  most  misleading  information  is  in  the  false  shape  identification  of 
Junctions  15  in  picture  1,  27  in  picture  2 (both  are  T Junctions  missing 
a line  and  reported  as  type  V)  and  2k  in  picture  2 (an  S Junction 
reported  as  type  V). 

The  program  reports  many  matchable  relations.  To  mention  a few: 
Junction  9 in  picture  1 is  reported  matchable  to  Junctions  1,8,10,12,27 
in  picture  2 and  to  Junction  21  in  picture  3.  Junction  27  in  picture  2 
(a  false  Junction)  is  reported  as  matchable  to  Junctions  5,9,10  in 
picture  1,  and  Junction  2k  in  picture  3 is  reported  as  matchable  to 
Junctions  5,6,16  in  picture  1 and  to  Junctions  1,7,11,17  in  picture  2. 
Thirteen  triples  are  formed  (the  subscript  in  the  Junction  indicates  the 
picture  number) 


i 
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(5l,l2,2'43) 

(3l’l62’63) 

(61,l2,2l*3) 

(161,172,2«»3) 


{9^,21^ 

(61’82’203) 
(11] ,92,193) 

(131,22,183) 


(lUlt32,173) 

(V132’V 

(ll’lU2’U3) 

(WV 

(101,102,23) 


On  the  basis  of  picture  context  the  last  nine  triples  can  be  accepted 
as  being  match  triples.  Of  the  four  rejected  triples,  two  are  the  three 
projections  of  two  vertices,  namely  (3^,l62,63)  and  (l6^,172,2U3) , but  at 
this  stage  no  context  support  is  found.  This  is  partly  corrected  in  the 
next  stage  where  we  compare  two  pictures  and  establish  matches  between 
pairs  of  Junctions.  In  the  first  pass  we  get  the  matches  of  the  pairs 
(^.lTg),  (r^.lSg),  (12^, Ug ) » (3^,162)  and  ( 7^ *13^ ) . In  the  second  pass 
we  get  (51,122),  (llg.l^).  (162,26^  and  The  last  one  gives, 

together  with  the  pair  reported  in  th*  first  pass,  the  match  triple 

(3^,l6g,63).  Hence  the  following  V Junctions  are  marked  as  valid: 

V 31*  V *l»  6i»  10i»  12i»  17i»  26i 

12.  3g»  7g.  102,  122.  132,  H»2,  162,  182 

13.  53,  63,  73.  133,  163,  173,  l83,  193,  213 
No  support  vas  gained  for  the  wrongly  reported  Junctions 


15j,  2*»g.  and  272 


nor  for  the  (correctly  reported)  Junctions 


5g»  3j  and  15j* 


IV. 


LINE  MATCHING  AND  DATA  RECOVERY 


I*.  1 Introduction 

Line  matching  is  not  a straightforward  procedure.  The  fact  that 
between  two  junctions  we  may  have  one,  two,  or  three  lines,  complicates 
the  situation,  and  it  becomes  even  worse  in  the  case  of  imperfect  data. 

We  have  to  be  very  careful  in  matching  lines  that  all  the  possibilities 
are  taken  into  consideration. 

Line  matching  also  serves  as  an  important  tool  for  recovering 
data.  Data  recovery  consists  basically  of  three  different  actions: 

(l)  detection  of  a missing  connection  between  two  Junctions,  (2)  exten- 
sion of  an  existing  line  from  one  valid  Junction  to  another  valid  Junc- 
tion, and  (3)  detection  of  the  location  of  a missing  valid  Junction, 
validating  the  one  found  there  (if  any),  or  creating  a synthetic  Junction. 

Data  recovery,  if  successful,  leads  to  conditions  for  further  line 
matching,  which  in  turn  enables  more  data  recovery,  and  so  on. 

Cyclic  order  in  an  unordered  valid  Junction  is  fixed  if  conditions 
for  Rule  1 or  Rule  2 exist  (See  Chapter  II).  Some  previously  unordered 
Junctions  can  be  ordered  on  the  basis  of  line  matching,  as  we  shall  see 
later.  This  in  turn  may  create  conditions  for  further  application  of 
Rule  1 or  Rule  2,  and  so  on. 

Line  matching,  data  recovery,  and  cyclic  ordering  is  done  in 
several  passes.  Each  pass  creates  new  conditions  for  the  next  pass  until 
finally  a condition  is  reached  in  which  no  new  results  can  be  obtaineo. 
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U.2  Fitting  :t  Line  to  a Set  of  Points 

It  is  important  for  the  procedure  to  be  able  to  determine  the  type 
of  line  represented  by  a set  of  points  and  sometimes  to  determine  its 
exact  equation  as  well. 

Given  a set  of  points  (x^z^),  we  try  to  fit  a line  to  it  so  that 
the  sum  of  the  squared  errors  will  be  minimized.  First  we  try  to  fit  a 
straight  line 


agX+ajZ+a  «0 


where 


V EVi  Evr  ^1 £z 


V £xi  £zr  £Vt  £xi 


a2=  Exi  Ez*-  Ezj  Ex^ 


2 

i 


The  average  distance  of  the  points  from  the  straight  line  is  calculated 
and,  if  it  is  less  than  some  threshold  T^,  we  accept  the  fit.  The 
threshold  is  selected  on  the  basis  of  visual  Judgment. 

If  the  average  distance  is  greater  than  some  T2>T^,  then  we  declare 
the  set  as  representing  a curved  line.  For  an  average  distance  lying 
between  these  two  values,  we  leave  the  type  undetermined.  If  the  set 
was  declared  to  represent  a curved  line  and  we  wish  to  fit  a conic  g 
to  it. 


2 p 

g(x,z)  = a^x  -t-a^ z +a^x  z+a^x+a^ z+aQ*0 
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(4  then  by  minimizing  the  sum  of  squared  errors 

_ 2,  . _ T „ 

E g (xi  ,zi)  = a *X*a 

and  constraining  aQ  to  equal  1,  we  can  solve  for  the  rest  of  the 
coefficients  using  X*a=0.  Here  a is  the  vector  of  coefficients  and 
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The  foregoing  methods  have  their  drawbacks  (11,16,1)  but  they  are  good 
enough  for  the  purpose  here. 

U.3  Line  Matching 

The  matching  of  lines  is  done  around  matched  Junctions,  taking  two 
pictures  at  a time.  Two  lines  in  two  pictures,  matching  the  same  line  in 
a third  picture,  are  declared  to  match  each  other.  With  every  Junction 
there  are  associated  three  indices  1,2,  and  3 for  the  three  possible 
lines  that  share  the  Junction.  (If  the  Junction  is  cyclically  ordered, 
these  indices  coincide  with  the  cyclic  order.)  We  say  that  the  Junction 
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is  linked  in  an  index  t if  there  is  a line  corresponding  to  that  index 
whose  other  end  Junction  is  a valid  Junction. 

The  two  matched  Junctions  around  which  we  try  to  match  lines  fall 
into  the  following  three  cases: 

(1)  both  contain  three  lines, 

(2)  one  contains  three  lines  and  the  other  contains  two  lines,  or 

(3)  both  contain  two  lines. 

In  each  of  these  three  cases  when  one  line  in  one  Junction  is  matched  to 
a line  in  the  other  Junction  and  both  Junctions  are  ordered  cyclically, 
further  line  matching  can  be  done  by  proceeding  cyclically  around  the 
Junctions. 

At  the  beginning  of  the  line  matching  process  all  the  3L  (3  lines) 
Junctions  are  cyclically  ordered  according  to  their  natural  order,  and, 
therefore,  in  Case  1,  matching  one  line  forces  the  match  of  the  other 
two  lines.  At  a later  step  in  the  procedure  a line  may  be  added  to  an 
unordered  2L  Junction,  yielding  an  unordered  3L  junction;  then  matching 
one  line  in  a Case-1  situation  will  not  automatically  lead  to  the  match 
of  the  rest. 

Fig.  4.1  shows  under  what  conditions  we  can  match  a line  in  Case  1. 
The  basic  idea  is  to  find  out  whether  between  a pair  of  Junctions  in 
the  same  picture  there  is  a single  line,  or,  if  more  than  one,  to  deter- 
mine their  relations  to  each  other.  For  example  in  configuration  (a) 
of  Fig.  4.1  it  is  clear  that  line  must  be  the  only  line  between 
Junctions  1^  and  because  is  linked  in  its  other  two  indices.  It 

follows  that  the  line  d.  is  the  only  line  between  Junctions  I and  J, 

J J J 


• 

valid  Junction 

relates  Junctions  that  are  already  matched. 

natural  extension. 

Legend 

© 

valid  Junction  that  is  already  cyclically 
ordered . 

a match  is  established  between  the  lines 

Fig.  I4.I  Case-1  Match  Configurations  (3-Llne  Junctions) 
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considering  the  pairwise  matching  of  the  Junctions  (though  it  might  not 

be  clear  from  looking  at  picture  J alone  when  J is  not  linked  in  its 

J 

two  other  indices).  Thus  the  result  is  a match  between  lines  c and  d . 

^ J 

Part  of  the  requirements  in  configurations  (c),  (e),  (f)  and  (h)  is  that 
the  Junctions  and  be  cyclically  ordered  to  establish  even  the 
first  match.  Conf igurations  (g),  and  (h)  require  some  special  condition, 
namely  that  1^  lie  on  the  natural  extension  of  the  line  c^.  This  means 
that  we  must  successfully  fit  a line  to  I,  and  the  points  of  c.  and 
that  this  line  comply  with  the  conditions  stated  in  the  following  para- 
graph. 

As  explained  in  section  h.2  a straight  line  fit  is  acceptable  if 
the  average  miss  distance  is  less  than  some  threshold.  If  a straight 
line  fit  is  rejected  and  a conic  is  fitted  instead,  we  get  either  an 
ellipse,  a hyperbola  or  no  locus  at  all.  Other  types  of  conics  are  not 
realistic  since  they  involve  a vanishing  parameter,  which  in  practice 
does  not  occur  in  the  computation  because  of  truncation  errors.  No 
locus  at  all  means  of  course  that  a conic  fit  has  been  rejected.  We 
also  reject  any  fit  accepted  thus  far  if, (l)  the  distance  between  I 
and  is  greater  than  the  distance  between  and  I (K^  is  the  non- 
valid  end  Junction  of  c^.),  (2)  if  it  is  an  ellipse  and  one  of  the 
axes  is  too  small  (to  avoid  a fit  "Jumping"  from  one  side  of  the 
ellipse  to  the  other),  (3)  if  it  is  an  ellipse  and  the  part  of  the 
ellipse  between  and  1^  is  greater  than  a quarter  of  the  ellipse,  and 
(U)  if  it  is  a hyperbola  and  the  points  of  the  set  belong  to  the  two 
branches  of  the  hyperbola.  This  tightening  of  acceptance  conditions 
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very  much  reduces  the  possibility  of  a wrong  extension,  but  at  the  same 
time  it  also  rules  out  some  good  extensions,  especially  under  condition 


(3). 

Configurations  c and  h present  a special  problem  since  they  both 
deal  with  Junctions  connected  by  three  lines.  To  obtain  matches  for 
them  we  assume  that  the  middle  line  stays  in  the  middle.  Although  true 
in  most  cases,  it  is  easy  to  show  counterexamples.  But  even  then  it  is 
not  crucial  since  the  configuration  is  closed  on  itself  and  the  error 
does  not  propagate.  To  find  the  middle  line  we  note  that  when  we  walk 
from  one  junction  to  the  other  and  back,  changing  lines  in  increasing 
cyclic  order,  we  trace  three  closed  paths.  In  two  of  them  the  path  is 
traced  in  a counterclockwise  manner  (Fig.  b.2),  and  in  one  of  them  in  a 
clockwise  manner.  The  last  one  happens  to  be  the  path  where  the  middle 
line  is  not  involved.  Using  this  fact  the  middle  lines  in  both  pictures 
can  be  detected  and  matched  to  each  other.  The  rest  follow  cyclically. 

Fig.  4.3  shows  the  configurations  required  for  matching  lines  in 
Case  2.  Note  configurations  ( c)  and  (j)  where  a natural  extension  is  a 
part  of  the  conditions.  Note  also  configurations  (d),  (k),  (e),  and 
(i),  which  imply  that  Junctions  J,^  and  1^  in  each  should  be  connected. 

We  shall  return  to  this  subject  in  the  next  section. 

Fig.  4.4  shows  the  conditions  under  which  we  match  lines  in  case 
3.  Note  in  both  configurations  the  implication  of  a missing  connection, 
between  and  1^  and  between  and  Kj  in  (a),  and  between  Jj  and  1^  in  (b) 
In  Figs.  4.1  and  4.3  the  shape  of  the  three-lines  Junctions,  (whether 
Y or  W)  is  irrelevant  to  the  configurations. 


Fig.  U.3  Case-2  Match  Configuration  (One  3-Line  and  One  2-Line  Junction) 


■ 


valid  Junction 


relates  Junctions  that  are  already  matched 


Legend 


a match  is  established  between  indicated 
lines 


Fig.  U.l»  Case-3  Natch  Configuration  (Two  2-Line  Junctions) 
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U . U Data  Recovery 

It  has  already  been  pointed  out  that  there  are  configurations  for 
which  it  is  clear  that  a connection  between  two  Junctions  is  missing. 

We  can  distinguish  between  two  groups  characterized  as  follows: 

(1)  Junction  J should  be  connected  through  its  line  c to  Junction  I, 
and  (2)  Junction  J should  be  connected  to  Junction  I but  the  line  of 
connection  is  missing  in  J.  The  action  to  be  taken  depends  on  the 
situation  in  Junction  I.  In  the  first  group  we  find  Case-1  configura- 
tions (c)ll,  (g),  (h)  (for  both  pictures),  (i)  and  (j),  and  Case-2 
configurations  (c),(d),  (i),  (j),  and  (k).  In  the  second  group  we 
find  Case-2  configurations  (e),  (h),  (i),  and  (j)  (between  Junctions 
and  K^),  and  Case-3  configurations  (a)  (for  both  pictures),  and  (b). 

In  the  second  group\we  find  also  configurations  (a),  (c),  and  (d), 

of  Case-2  whenever  the  valid  end  Junction  of  the  only  unmatched  line 

« \ 

has  a matched  Junction  in  the  other  picture. 

In  the  first  group  we  klso  find  the  case  where  a line  d^  matches 
a line  d^  whose  both  end  Junctions  Ij  and  Jj  are  valid  and  match  Junc- 
tions 1^  and  J^,  but  only  belongs  to  d^.  See  Fig.  b.5(a).  It 
occurs  mostly  in  Cases  1 or  2 when  lines  are  matched  around  the  cycle, 
after  one  match  was  established.  \ 

Table  U.l  shows  us  the  different  possibilities  and  appropriate 
actions  taken  for  the  first  group  anil  Table  k.2  does  the  same  for  the 
second  group.  In  both  cases  it  is  asaWed  that  the  index  in  J with  which 
we  associate  the  line  of  connection  is  r\  From  the  nature  of  the 
different  configurations,  r is  always  knowi.  The  problem  is  to  determine 
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Table  U.l 

Conditions  for  First-Group  Junction  Connections 


I is  a 2L  Junction  and  linked  in  indices  r and 
s.  Action:  Extend  line  c to  junction  I and 

associate  it  with  index  t in  I.  I becomes  a 3L 
Junction,  linked  in  all  three  indices.  J is 
linked  in  index  r. 


I is  a 3L  Junction  linked  in  indices  r and  s. 
Action:  The  line  associated  with  index  t in  I 

is  combined  with  line  c.  I becomes  linked  in 
three  indices.  J is  linked  in  index  r. 


r_ 

I is  a 3L  Junction  linked  in  index  s.  Action: 
The  average  error  of  points  on  the  line  assoc i- 

y 

ated  with  r and  t is  calculated  with  respect  to 
the  natural  extension  of  c.  The  line  with  the 
smaller  average  error  is  combined  with  c . I 

becomes  linked  in  a second  index.  J becomes 
linked  in  r. 

I is  a 1L  Junction  linked  in  index  s.  Action: 
Extend  line  c to  Junction  I and  associate  it  with 
index  t or  r.  I becomes  a 2L  junction,  linked 
in  two  indices.  J is  linked  in  index  r. 


I is  a 2L  Junction  linked  in  index  r.  Action: 
Calculate  the  average  error  of  the  points  on  the 
line  associated  with  t,  with  respect  to  the 
natural  extension  of  c.  If  the  error  is  less  than 
some  threshold  combine  the  two  lines.  I becomes 
linked  in  t and  I in  r. 


Other 


No  Action 


Picture  1 


Picture  J 


Fl«.  M Data  Recovery  Situation 
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with  which  index  in  I to  associate  this  line.  The  possibility  of 
having  a 1L  valid  Junction  will  be  demonstrated  later  in  this  section. 

It  is  a result  of  certain  data  recovery  actions. 

By  "empty"  line  mentioned  twice  in  Table  h.2,  we  mean  a line  for 
which  only  the  end  points  are  known  (the  two  Junctions)  and  nothing 
is  known  about  its  nature.  Its  importance  lies  in  creating  the  links, 
which  in  turn  create  new  conditions  for  matching  lines  around  I in  the 
next  pass.  The  linking  of  I and  J may,  for  example,  qualify  Junction  I 
for  analysis  as  one  of  the  Case-1  configurations  and  may  enable  the 
application  of  Rules  1 and  2.  We  shall  return  to  the  detection  and 
completion  of  missing  connections  once  again  in  what  we  call  second-level 
data  recovery. 

Let  us  consider  the  situation  shown  in  Fig.  J*.5(b).  Lines  di  and 

dj  either  match  each  other  or  are  candidates  for  a match.  But  there  is 

no  match  in  picture  i for  Junction  I . This  might  be  crucial  for  the 

J 

matching  decision  for  the  lines  d^  and  d^  as  we  can  see  by  comparing 

Fig.  U . 3 (b ) with  Fig.  l*.3(c).  If  there  is  for  I.  a matching  Junction  I 

J K 

in  the  third  picture,  it  could  serve  as  a basis  for  further  tests  and 
data  recovery.  We  may  then  have  two  different  scenarios: 

1.  There  exist  a Junction  1^  in  picture  i,  unmatched  as  yet,  that  is 
matchable  to  1^  and  to  1^.  If  1^  lies  on  the  natural  extension  of 
dj,  then  (1^,  1^,  1^)  is  declared  as  a match  triple,  and  we  conclude 
that  the  Junctions  and  1^  should  be  connected.  We  also  match  the 


lines  di  and  d^  if  they  are  not  matched  as  yet. 


i 
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TABLE  k.2 

Conditions  for  Second-Group  Junction  Connections 


I is  a 2L  Junction  linked  in  r and  s.  Action: 
Create  a new  "empty"  line  between  I and  J. 
Associate  it  with  index  t in  I and  index  r in  J. 

I becomes  a 3L  Junction  linked  in  all  indices. 

The  number  of  lines  in  J is  increased  by  one  and 
J is  linked  in  index  r. 

I is  a 3L  Junction  linked  in  2 indices  r and  s. 
Action:  Extend  the  line  associated  with  t from 
I to  J and  associate  it  with  index  r in  J.  I is 
linked  now  in  three  indices.  The  number  of  lines 
in  J is  increased  by  1.  J is  linked  in  index  r. 

I is  a 3L  Junction  linked  in  index  t.  Action: 

Find  which  of  the  two  lines  associated  with 
indices  r and  s has  a natural  extension  to  J. 

If  there  is  one,  extend  the  corresponding  line 
to  J.  I is  linked  then  in  two  indices.  Number 
of  lines  in  J is  increased  by  1,  and  it  is 
linked  in  index  r. 

I is  a 1L  Junction  linked  in  index  t.  Action: 
Create  an  "empty"  line  between  I and  J.  Associ- 
ate the  line  with  index  r in  J and  with  one  of 
the  indices  r or  s in  I.  I becomes  a 2L  Junction 
linked  in  two  indices.  The  number  of  lines  in  J 
is  increased  by  one,  and  J is  linked  in  index  r. 

I is  a 2L  Junction  linked  in  index  r.  Action: 
Check  whether  J is  on  the  natural  extension  of  the 
line  associated  with  t.  If  so,  extend  this  line 
to  J.  I is  now  linked  in  two  indices.  The  number 
of  lines  in  J is  increased  by  1 and  it  is  linked 
in  its  index  r. 


Other 


No  Action 


G 
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2.  If  there  is  no  Junction  I.  that  is  matchable  to  I and  I , (and 

J k 

has  not  been  matched  as  yet)  then  we  build  a synthetic  Junction. 

This  is  done  as  follows:  From  I and  I,  we  can  determine  the 

J k 

approximate  location  in  3D  of  the  corresponding  vertex.  It  will  be 
the  point  with  the  minimal  sum  of  absolute  distances  from  the  pro- 
jecting lines  1^  and  Cj  Ij . This  point  is  found  most  easily  by 
looking  at  the  x,  y coordinates  of  the  intersection  of  the  lines' 
projections  on  an  x-y  plane  in  a coordinate  system  whose  z axis  is 
normal  to  both  lines.  The  calculated  projection,  from  on  picture  i, 
of  this  vertex  will  serve  as  the  synthetic  Junction  1^.  If  the  lines 
dt  and  dj  are  not  matched  as  yet  but  the  synthetic  Junction  ^ lies 
on  the  natural  extension  of  d^,  or  if  d^^  and  dj  already  match,  then 
we  accept  the  synthetic  Junction  as  a new  valid  Junction,  declare  the 
triple  (1^,  1^ , 1^)  as  a match  triple,  and  extend  the  line  d^  to  1^. 

1^  is  now  a valid  1L  Junction  linked  in  one  index. 

To  demonstrate  how  important  a synthetic-junction  formation  can  be 
for  line  matching,  let  us  look  at  the  three  images  shown  in  Fig.  14.6(a). 
Fig.  14.6(b)  shows  the  line  matching  that  can  be  established  before  the 
formation  of  the  synthetic  Junction  Uj.  Fig.  U . 6 ( c ) shows  the  line 
matching  established  by  the  configuration  of  Fig.  U . 1 ( i ) , possible  only 
after  was  formed. 

Finally,  an  unordered  Junction  for  which  Rules  1 and  2 cannot  be 
applied,  can  be  ordered  if  at  least  two  of  its  lines  are  matched  to  two 
lines  of  an  ordered  Junction  so  that  the  cyclic  precedence  of  the  first 
two  will  agree  vith  that  of  the  last  two.  The  indices  associated  with 
the  formerly  unordered  Junction  are  then  adjusted  to  comply  with  the 
cyclic  order. 


'•.TW7 S7- ' 
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4.5  Example 

We  shall  describe  some  of  the  results  obtained  when  a program  for 
the  foregoing  procedure  was  tested  on  the  scene  shown  in  Fig's.  3.6,  3.7 
and  3.8.  In  Fig.  4.7  we  give  a schematic  description  of  the  three 
pictures.  First  the  unordered  Junctions  6^  12^  17^,  lg,  32>  and 
lS^  were  ordered  each  by  applying  Rule  1. 

The  configuration  of  Fig.  4.3(a)  in  14^  and  32  brought  the  match 
of  lines  12^  ll^  to  , 3g  and  13^7TI*]"  to  32»  2g. 

The  configuration  of  Fig.  4.3(a)  in  14^  and  173  brought  the  match 
lines  14^,  13^  to  17^,  18^  and  l4^ , 2 6^  to  17^,  l63»  Combining 
thi*  with  the  preceding  match  we  get  the  match  17^,  18^  to  3^,  2g. 
Through  this  match  we  are  able  also  to  arrange  the  cyclic  order  in 
Junction  17^. 

The  configuration  of  Fig.  4.1(a)  in  11  and  92  gave  the  match  of 
lines  111#  6X  to  9^7"5J  , llj.,  13j  to  9g.  2g  and  11  , 12  to 


The  configuration  of  Fig.  4.3(a)  in  1^  and  193  gave  the  match  of 
lines  11^,  6^  to  193»  203  which  will  match  92»  8g  - combining  the 
above  - and  11^,  13^  to  193>  l83  which  will  match  9g,  6g.  The 
result  of  the  matching  of  the  last  group  is  that  we  can  now  establish 
the  cyclic  order  in  Junction  193< 

Also  6^,  9^  match  8g,  lg  based  on  the  configuration  of  Fig.  4.3(a) 
in  6^^  and  8g  ; match  203 , 213  on  basis  of  Fig.  4.3(a)  in  61  and 

203  ; and  combining  the  two  we  get  that  203»  213  matches  82»  1^. 


PICTURE  1 


3 


PICTURE 


PICTURE  3 


Fig.  U 


7 Scheswtic  Description  of  the  Pictures  Shorn 
in  Pig's.  3.6,  3.7  and  3.8. 
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From  Fig.  4.1(a)  in  13.^  and  2^  we  obtained  the  match  of  9^ . 13.^ 
to  2^,  lg»  From  Fig.  4.3(a)  in  13^  and  18^  and  a match  between  Junctions 
9^  and  21^  we  obtain  the  linkage  of  Junctions  18^  and  21^  by  an  "empty" 
line,  which  was  matched  to  line  13^  91  and  thus  to  line  2 2>  12« 

Junction  18^  was  arranged  cyclically  by  comparison  with  Junction  13^, 
and  this  enabled  the  application  of  Rule  1 for  ordering  Junction  21^; 
this  in  turn  enabled  the  application  of  Rule  1 for  ordering  Junction 

133* 

Next  we  obtained  also  the  match  of  line  9^  15.^  to  21^,  lU^  from 
the  Just  formed  Fig.  4.1(a)  configuration  in  Junctions  9.^  and  21^. 

From  4.3(a)  in  1^  and  4^  we  found  the  match  of  1^,  4^  to  4^,  7 . 

The  Fig.  4.1(a)  configuration  in  2.^  and  152  gave  the  match  of  2 1 , 1 

to  152,  142,  2^ , 10^  to  152,  lOg  and  2.,  31  to  152,  1^2-  The  configura- 
tion of  Fig.  4.3(a)  in  2^  and  5^»  together  with  the  match  of  3^  and  6^ 
brought  the  formation  of  an  "empty"  line  between  5^  and  6^,  its  match 

to  line  2^ , 3^^  (and  thus  to  152,  l62),  the  match  of  53»  2^  to  2.^,  101 

(and  thus  to  152*  lOg),  the  match  of  5^,  4^  to  2^,  1^  (and  thuB  to 
15g»  l**2)t  and  cyclic  ordering  of  Junction  5^. 

Line  82»  72  was  matched  to  20^,  13^  using  the  Fig.  4.1(a)  config- 
uration in  8g  and  20^.  Line  llg,  102  was  matched  to  1^~2^»  and 
112,  162  to  1^»  6^  using  the  Fig.  4.3(a)  configuration  in  112  and  1^. 
Junction  13  is  ordered  by  comparing  it  with  Junction  112-  This,  together 
with  the  last  paragraph  results,  permits  the  ordering  of  6 3 by  Rule  1. 

The  configuration  of  Fig.  4.4(b)  in  132  and  ?3  permits  132  and  l6?  to  be 
connected  by  an  "empty"  line  which  is  matched  to  line  7 3>  63;  line  132,  l42 
is  matched  to  7^7"^  and  thus  also  to  line  1^  4^ 
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In  Junctions  and  13g  the  configuration  of  Fig.  U.3(a)  was  formed 
only  after  the  connecting  of  Junctions  13^  and  16,,.  Otherwise  no 
qualified  configuration  could  have  been  found  there.  This  configuration 
brings  the  match  of  line  U^,  5^  to  line  132,  12,,. 

We  also  tested  for  the  configuration  of  Fig.  U.l(h)  in  Junctions 
16^  and  172«  (in  the  beginning  of  the  procedure  we  merge  the  two  non- 
limb lines  in  every  S Junction  into  one  line.)  The  extension  of  line 
lT^*  20,,  to  Junction  18^  was  rejected  on  the  basis  of  condition*  3. 

But  a natural  extension  of  line  l6^,  19-^  to  Junction  17.^  was  found,  quali- 
fying the  configuration.  Thus  line  16^,  17^  was  matched  to  line 
172,  182,  line  I^T^l  was  extended  to  Junction  17^  and  matched  to  line 
172»  which  is  now  extended  on  the  basis  of  the  match  to  Junction  l82- 

Finally,  line  16^,  18  , 17^  was  matched  to  line  172,  19,,,  l82. 

These  matches  were  reached  before  making  use  of  any  synthetic 
Junctions.  Let  us  see  how  the  matching  ability  is  improved  by  them. 

As  we  can  see,  line  3^,  8 could  not  be  matched.  If  the  configuration 
found  in  3^  and  6^  (Fig.  U.3(b))  could  be  modified  to  the  configuration 
of  Fig.  1* . 3(c ) then  the  desired  match  could  be  established.  (Junction  ^ 
failed  to  be  on  the  natural  extension  of  3^»  8^).  Hence,  Junctions 
112  and  1^  are  used  to  create  a synthetic  Junction  in  picture  1 which  we 
denote  as  28^.  Junction  28^  is  found  to  be  on  the  natural  extension  of 
3^7  8^  and  thus  is  accepted  as  a valid  Junction.  Line  3^,  8^  is  extended 


* See  discussion  concerning  natural  extension  in  Section  4.3 


1 
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to  be  line  3^ , 28  ^ and  the  triple  (28^,  11^,  1^)  is  declared  to  be  a 
match  triple. 

The  configuration  of  Junctions  11^  and  1^  force  the  connections  of 
Junction  28^.  Thus  Junction  28^  should  be  connected  to  Junctions  10^ 
and  5^  under  the  rules  of  Table  4.2.  It  is  found  to  be  on  the  natural 
extension  of  10^,  15^  which  is  thus  extended  to  be  line  10^ , 28^.  It  is 
also  found  to  be  on  the  natural  extension  of  line  5^  7-^  which  turns  to 
be  line  5^»  28^.  The  new  configuration  created  now  in  Junctions  3^  and 
and  6^  (of  form  Fig.  4.3(e)),  causes  the  match  of  3^7  28^  to  ^ , 1^  and 
thus  also  to  l62,  112»  as  well  as  the  connection  with  an  "empty"  line  of 
3^  to  4^  (and  its  match  to  6^,  7^  and  thus  also  to  the  "empty"  line 
132>  l62).  We  also  get  the  match  of  10^,  28^  to  1^,  2^  (configuration  of. 
Fig.  4.3(a)  in  28^  and  1^)  and  thus  also  to  10^,  llg.  In  addition  we 
get  the  match  of  5^,  28^  to  122>  112  (configuration  of  Fig.  4.1(a)  in  28  ^ 
and  llg),  and  finally  the  following  Junctions  are  ordered  cyclically  as 
the  result  of  all  this  additional  matching  and  linking:  1^,  3^,  4^,  5^»  10^, 
28v  102,  122,  132,  142,  162  and 

4.6  Matching  Lines  by  Range 

The  technique  of  matching  lines  around  matched  Junctions  cannot  be 
applied  when  the  lines  are  closed,  when  they  have  end  Junctions  that  are 
both  of  type  A,  when  the  ends  are  both  open,  or  when  one  end  Junction  is 
of  type  A and  the  other  end  is  open.  Additional  tools  are  needed  for 
establishing  a match  in  these  situations.  We  define  for  a line  g^  in 
picture  i range  limits  with  respect  to  picture  .1  as  the  two  straight  lines. 
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I 


with  the  maximum  angle  of  opening,  going  through  C and  having  points  in 

1 J 

common  with  g^.  (Lines  a and  b in  Fig.  U . 8 ) . For  a closed  line  the  range 


limits  are  the  two  tangents  to  the  line  from  C 


ij' 


Taking  two  pictures  i and  j,  we  create  in  each  picture,  for  all  quali- 


fied lines,  pairs  of  range  limits  with  respect  to  the  other  picture.  If 

we  now  pioject  both  pictures  on  a plane  (as  explained  in  Chapter  III  and 

shown  in  Fig.  3.2)  we  get  all  the  range  limits  from  both  pictures  as  a set 

of  lines  going  through  If  we  take  the  range  limits  of  a line  g^, 

then  when  projected  on  P^,  they  form  an  angle  a.  The  range  limits  of  the 

line  Ik  when  reprojected  on  P^  form  the  angle  0.  We  define  a ratio 

Rt  = a+6  where  y is  the  overlap  angle  (See  Fig.  1*.8).  When  R^  is 

bigger  than  some  threshold,  we  say  that  g and  h are  matchable  by  range 

^ J 

(but  are  not  matched  yet).  When  we  have  three  lines  in  three  pictures, 

lines  g.,  h.  and  f > pairwise  matchable  by  range,  then  we  say  that  the  lines 
l J n 

are  matched  by  range.  In  our  example  the  lines  23^»  22^  , 21^,  2 6^  and 
10^,  9^,  among  others,  are  matched  by  range. 


V.  FINAL  DESCRIPTION 


5.1  Region  Assembly  and  Second-Level  Data  Recovery 

In  the  region  assembly  phase  we  try  to  find,  in  all  three  pictures 
simultaneously,  all  the  lines  that  correspond  to  the  same  boundary  or  - if 
a face  has  more  than  one  boundary  - to  the  boundaries  of  the  same  face. 
Each  line  should  be  assembled  twice,  since  its  corresponding  edge  is 
shared  between  two  faces.  By  face  group  we  denote  the  collection  of  lines 
from  three  pictures  that  correspond  to  the  boundaries  of  the  same  face. 

Whenever  a line  is  found  to  belong  to  a certain  face  group,  the 
lines  that  match  it  in  the  other  pictures  are  also  added  to  this  face 
group.  We  build  a face  group  by  selecting  a line,  in  any  picture,  that 
was  not  assembled  yet  in  at  least  one  direction.  We  then  follow  the  line 
assembly  (see  definition  in  Chapter  II),  in  the  picture  in  this  direction, 
until  we  either  (l)  close  the  line  assembly  - i.e.,  reach  the  line  we 
started  with,  or  (2)  no  line  can  be  found  for  further  assembly  in  this 
picture.  If  the  assembly  is  terminated  because  of  (l)  then  one  boundary 
was  completely  traced.  When  we  terminate  because  of  (2),  then  we  select 
a line  in  another  picture  matching  the  last  assembled  line.  If  such  a 
line  is  found,  then  we  repeat  the  whole  procedure,  and  so  on.  Fig.  5.1 
shows  how  the  assembly  1b  carried  out,  "Jumping"  from  one  picture  to 
another.  If  the  assembly  process  is  blocked  in  all  three  pictures,  we 
return  to  the  starting  point  and  trace  the  lines  in  the  opposite  direction, 
changing  lines  at  the  Junctions  in  decreasing  cyclic  order  and  following 
the  above-described  procedure  until  we  are  blocked  again  in  all  three 
pictures.  Although  we  are  now  tracing  the  lines  in  the  opposite  direction. 
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PICTURE  i 


Fig*  5*1  Face  Group  Assenbly 
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we  record  the  trace  as  if  it  were  done  in  the  right  direction,  and  a 
change  from  line  c^  to  line  d^  in  the  decreasing  cyclic  order  is  recorded 
as  a change  from  line  d3  to  line  Cj  in  the  increasing  cyclic  order. 

For  example  consider  the  scene  shown  in  Fig.  U . 7 . The  line  11  , 6^ 
was  selected  as  the  first  element  of  a new  face  group,  going  fran 
111  t0  61*  ItS  matching  lines  92>  82  and  193,  20 ^ were  then  added  to  the 
face  group.  A line  next  in  cycle  was  not  found  in  6^  and  hence  the  line 
next  in  cycle  in  &2  was  picked  - line  82,  its  matching  line  20^,  133 

was  also  added  to  the  face  group.  Here  the  assembly  was  blocked  since 
neither  72  nor  133  had  a line  next  in  cycle.  Trying  to  extend  the 
assembly  in  the  opposite  direction,  beginning  from  the  starting  lines,  we 
are  able  to  proceed  in  both  11^  and  92*  Next  to  be  assembled  then  was 
line  11^,  12^  and  its  matching  line  92«  **2-  Since  no  next  line  in  the 
decreasing  cyclic  order  was  found  in  12^ , the  next  line  in  the  decreasing 
order  in  namely  lines  l*^,  52*  was  picked.  The  assembly  terminates  here 
since  52*  having  no  match,  was  not  validated  and  not  ordered.  The  record- 
ing of  this  face  group  assembly  is  done  as  follows: 


Let  us  consider 
pletely  traced.  The 


another  example,  in  which  a closed  boundary  is  corn- 
first  line  picked  for  this  face  group  is  1 , 1*^. 
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(The  stars  indicate  the  places  where  the  linkage  between  the  two  Junc- 
tions is  the  result  of  the  data  recovery  phase): 


v"1!  ■ 

*iF5r ' 

, *132.  162  t3.  «3 

V 2i 

' FFFt  'Fh 

21‘  X1 

The  existence  of  a virtual  Junction  usually  indicates  the  existence  of  a 
multiple-boundaries  face.  Thus  on  assembling  the  lines  that  belong  to  a 
single  boundary,  all  S and  A Junctions  are  remembered*;  when  the  boundary 
trace  is  terminated,  we  return  to  these  Junctions,  search  for  the  other 
end  of  its  limb,  and  start  there  another  line  assembly  for  the  same  face 
group.  The  limbs  themselves  are  also  added  to  this  face  group. 

When  all  lines  have  been  twice  assembled,  we  may  try  some  further 
data  recovery  actions.  This  second-level  data  recovery  is  carried  out  in 
the  three  following  configurations  shown  in  Fig.  5.2. 

1.  Two  lines  1 and  k in  the  same  picture,  with  one  non-valid  end 

Junction  in  each,  are  assembled  into  two  face  groups  M and  N, 
k,l  e M and  k,lcN.  Action:  Combine  1 and  k into  one  line. 

2.  A line  1 with  one  non-valid  end  Junction  is  assembled  into  two 

face  groups  M and  R,  and  there  is  in  the  same  picture  a 2-lines 
Junction  J with  lines  ■ and  n,  meM  and  neR.  Action:  Extend 

the  line  1 to  Junction  J. 

• In  Chapter  III  it  was  stated  that  the  two  non-limb  lines  in  every  S 
Junction  are  merged  into  one  line.  Thus,  on  assembling  the  lines, 
there  is  no  change  of  lines  in  an  S Junction;  the  Junction  is  bypassed 

and  remembered. 


— -- rr-- -- 
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3.  There  are  two  2-lines  Junctions  I and  J in  the  same  picture 
with  lines  k,l  and  h,g,  respectively,  such  that  k and  h are 
assembled  into  face  group  M,  and  1 and  g are  assembled  into 
face  group  N.  Action:  Create  an  "empty"  line  between  the  two 

a 

Junctions  I and  J. 

The  data  recovery  actions  utilize  the  fact  that  two  components 
assembled  simultaneously  into  two  face  groups  must  correspond  to  the 
intersection  of  the  two  corresponding  faces.  There  is  of  course  the 
possibility  that  this  intersection  is  sectioned  into  several  separate 
edges  and  that  the  two  components  correspond  to  two  different  sections; 
but  at  this  stage  of  the  recognition  phase,  with  no  evidence  to  the 
contrary,  we  assume  that  they  belong  to  the  same  section. 

After  any  second-level  data  recovery  actions  have  been  performed, 
we  try  again  to  match  lines,  using  the  new  conditions.  Then  conditions 
for  second-level  data  recovery  are  checked  again,  and  so  on  until  no 

» 

further  line  matching  is  possible.  The  face-group  set  is  updated  each 

I 

time  nev  results  are  obtained. 

5.2  Object  Formation 

Each  of  the  face  groups  defines  (corresponds  to)  a face  of  one  of 
the  bodies  in  the  scene.  After  the  face-group  assembly  and  second-level 
data  recovery  is  completed,  we  are  ready  for  the  formation  of  objects. 
Each  object  is  a set  of  face  groups,  vith  no  cosmion  elements  betveen  two 

» 

sets.  Every  face  group  must  belong  to  some  set.  A new  object  is  formed 
by  picking  a face  group  that  is  still  free,  then  recursively  adding  to 
the  set  every  face  group,  not  as  yet  in  the  set,  which  has  a line  in 
coanon  vith  any  of  the  face  groups  already  in  the  set. 
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5.3  Type  of  Face  Determination 

We  would  like  to  know  for  each  face  group  whether  it  defines  a 
planar  or  curved  face.  Toward  that  end  let  us  define  the  predicate  C(A) 
for  every  face  group  A as  "the  face  defined  by  A is  curved".  When- 
ever a face  group  has  a limb  in  it,  we  have,  by  definition,  C(A)=1. 

When  two  face  groups  A and  B share  a straight  line*,  no  information  is 
conveyed  about  the  faces'  nature.  However,  when  they  share  a curved 
line,  we  must  have  C(A)  + C(B)  * 1,  that  is,  one  of  the  faces  or  both 
must  be  curved.  Thus,  we  have  for  every  object  a set  of  logical  equa- 
tions formalizing  the  constraints  imposed  by  the  types  of  lines. 

The  philosophy  underlying  our  strategy  for  determining  the  nature 
of  faces  is  the  following:  Given  several  pictures,  every  curved  face 

will  somehow  manifest  itself  or  there  is  no  way  to  recognize  it  as 
curved.  Thus,  we  are  looking  for  a minimal  cover  to  satisfy  the  set  of 
logical  equations  for  every  object  based  on  the  information  collected 
from  all  three  (or  more)  pictures.  This  minimal  cover  will  determine 
which  of  the  face  groups  define  a curved  face. 

We  find  the  minimal  cover  by  means  of  a minimum-cost  tree  search, 
where  the  cost  function  is  the  number  of  faces  thus  far  determined  as 
curved.  Consider  the  one-picture  example  shovn  in  Fig.  5.3.  The  set 


* In  Chapter  IV  ve  discussed  how  we  determine  whether  a line  is  straight 
or  curved.  We  say  that  two  face  groups  share  a curved  line,  if  in 
at  least  one  picture  there  is  a curved  line  which  they  share. 
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ot equations  formalizing  the  lines  constraints  is: 

1.  C(B)*1 
•2.  C(B)+C(A)»1 
•3.  C(B)+C(F)*1 

c(b)+c(e)=i 

5.  c(c)+c(e)=i 

6.  c(a)+c(c )*1 
•7.  C(B)+C(D)*1 

8.  c(f)+c(c)*i 

9.  C(C)+C(D)=1 

•10.  c(g)+c(b)=i 

The  tree  search*  beginning  with  the  first  equation,  is  given  in 
Fig.  5.U.  The  starred  (•)  equations  are  those  that  give  no 
additional  constraint  since  one  of  their  elements  participates 
in  a single-member  equation  (e.g.,  B).  The  minimum-cost  tree  search 
gave  the  cover  that  determined  the  faces  corresponding  to  B and  C as 
curved.  (Which  agrees  with  our  intuitive  interpretation  of  the  picture). 
After  the  nature  of  the  faces  is  determined  ve  may  wish  to  classify 
those  that  are  curved  as  to  the  type  of  their  quadric  equation.  The 
flowchart  of  Fig.  5.5  shows  a way  of  doing  this. 

5.h  Face  Equations 

Ve  have  already  explained  how  the  approximate  3D  location  of  a 
point  is  determined  from  its  two  projections  by  taking  the  point  of  minimum 
sua  of  distances  from  the  two  projecting  lines.  If  we  are  given  three 
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(l) 

*(2) 

»(3) 

*(M 

(5) 


(6) 

»(7) 

(8) 

(9) 

•do) 


Fig.  5.3  Tree  Search  for  Minimal  Cover  to  Determine  Curved- 
Face  Group.  (The  number  of  the  equation  yielding 
each  particular  nev  level  of  nodes  is  indicated 
at  the  right . ) 
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Yes 


Does  the  face  group 
have  a limb? 


Are  there  two  straight  limb 
lines  in  the  same  picture 
belonging  to  this  face  group? 


No 


Is  the  limb  an 
ellipse? 


> /Yes 

^elMjasoic^j  No 


Yes 


Yes 


Are  they 
parallel 


Yes 


a 


No 


Are  there  two  lines  in  the  facp 
group,  which  are  determined  as 
straight*,  in  the  same  picture? 


4/ No 


cylinder  |cone 
1 Is  it  hyperbola? 


1 

Are  they  parallel? 

vJ/Yes 

\ 

L No 

cylinder 

cone 

No 


Unable 

to 

classify 


Yes 


No 


Unable  to 
classify 


Yes 


Make  this  check:  Take  a point  x^y^  on  a line  in  the 


same  face  group  in  the  same  picture, 
the  hyperbola  equation.  Is 
2 2 

sign  of  a5x1  + a,^  + a„x,z,  + a„x 


Substitute  in 


3 11 


2-1  + Vl  + a0  = 


r-rNo 


sign  of  a^*a^*aQ? 


Hyperboloid  of  one 

Hyperboloid  of  two 

sheet 

sheets 

• Note:  straight  lines  exist  also  on  other  types  of  quadric  surfaces 
but  we  assume  the  most  likely  possibility.  It  is  also  assumed  that 
parallelization  is  not  badly  distorted. 


Fig.  5.5  Flow  Chart  for  Determination  of  Type  of  Quadric  Surface 
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projections  of  a point,  then  for  a better  approximation  we  may  determine 
three  3D  locations  (by  taking  two  projecting  lines  at  a time)  and 
then  take  their  mean.  This  will  give  a very  good  approximation  of  the 
3D  location. 

If  we  wish  to  have  the  equation  of  a face  which  we  know  to  be 
planar,  all  its  vertices  which  have  at  least  two  matched  images  can  be 
collected  in  3D  and  a plane  equation  fitted  to  them  by  minimizing  the 
sum  of  square  errors. 

We  shall  now  show  how  the  equation  of  a curved  face  can  be  deter- 
mined from  its  limb  line  in  three  different  pictures. 

The  equation  of  a quadric  surface  is 

T 

Q(x,y,z)  = u *A*u  = 0 
where  A is  the  matrix  of  coefficients 


all 

*12 

*13 

*11* 

*12 

a22 

*23 

*21. 

*13 

*23 

a33 

*3U 

*11* 

*21* 

*31* 

al*l* 

and  u is  a point  in  homogeneous  coordinates 

T 

u = (x,y,z,l) 

If  we  denote  by  u^  the  center  of  projection  for  picture  i,  then  the 
limb  of  Q with  respect  to  u^  lies  on  the  polar  plane  of  u^  with  respect 

T 

P(x,y,z)  ■ u^  *A*u  ■ 0 


to  Q. 
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The  cone  of  projection  belongs  to  the  linear  family 

T rn  p 

jj  *A*u  -k(u_.*A*u)  = 0 

Since  belongs  to  the  cone  of  projection,  we  get 


k = 


1 


T A 

u. »A«u. 

— l 1 


and  the  cone  of  projection  for  the  surface  onto  picture  i is 

T T T 2 

c(x,y,z)  = ^‘A'u^*!!  »A*u_  - (u^*A»u)  = 0 

Expressing  this  cone  in  picture  i's  coordinate  system  and  substituting 

0 for  y*  coordinate,  we  get  the  equation  of  the  limb  expressed  in  terms  of 

the  unknown  quadric's  coefficients.  The  coefficients  of  the  calculated 

limb  line,  which  are  functions  of  the  desired  coefficients  a .,  should 

n,j 

be  proportional  to  the  coefficients  of  the  fitted  conic.  This  gives 
us,  for  ideal  data,  six  non-linear  equations  in  the  ten  unknowns  plus 
the  unknown  proportionality  coefficient: 


*12- 


*33’  aW  °i>  * 0 


n*l. . . .6 


A second  picture  will  give  us  six  more  equations  but  only  four  of  them 
will  be  independent  of  the  first  picture;  it  will  add,  however,  an 
unknown  proportionality  coefficient. 


*33*  mM»*  CJ 


) 


••4. ...4 
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The  third  picture  will  add  six  more  equations  and  another  unknown 


Pn,k(,U-  a12" 


•*33’  V’  ck)  ' 0 


n=l ....  6 


but  only  two  of  the  equations  will  be  independent  of  the  other  two 
pictures.  This,  together  with  the  normalization  condition 


1 


give  us  thirteen  non-linear  equations  in  thirteen  unknowns.  In  the 
case  of  real  data  we  can  solve  for  the  unknowns  by  looking  for  a 
minimum  for  the  functions  F together  with  the  last  equality  equation. 
The  algorithms  concerning  the  faces'  equations  were  not  programmed. 


VI.  EXPERIMENTAL  RESULTS 


6.1  First  Example 

For  the  first  example  consider  the  scene  which  was  discussed  in 


Chapters  3 and  1*  (see  Fig. 

1*.7). 

The  face 

groups  reported  by  the  pro- 

gram  are  as  follows: 

(For  line  notation  see  Fig.  6.1,  where  also  part 

of  the  data  recovery 

results  are  i 

3hown . ) 

Pic.l 

Pic.  2 

Pic.  3 

Pic.l  Pic. 2 Pic. 3 

Face  Group  1 

Lines: 

33 

26 

28 

Limb  Lines: 

31* 

28 

26 

Face  Group  2 

Lines: 

33 

26 

28 

Face  Group  3 

Lines: 

31 

2k 

10 

23  21  13 

Limb  lines: 

22,21 

23,25 

ll*,ll 

Face  Group  4 

Lines: 

31 

2l* 

10 

Face  Group  5 

Lines: 

30 

31 

- 

25 

29 

•- 

Limb  lines: 

27 

33 

Face  Group  6 

Lines: 

30 

31 

- 

29  31* 

Face  Group  7 

Lines : 

29 

3U 

- 

25  29 

Face  Group  8 

Lines : 

23 

21 

13 

Face  Group  9 

Lines: 

20 

- 

19 

13  13 

- 

lU 

- 

7k  - 

75  - 


PICTURE 


'ICTTJP.E 


PICTURE 


rig.  6.1 


First  ExsBpls.  (The  Circled  Junctions  are  Cyclically 
Arranged.  The  Arrows  Point  at  Data  Recovery  Results. 
The  Broken  Lines  Indicate  "fcpty  Li net",  for  which  only 
the  Rod  Points  are  Known. ) 
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Pic.l 

Pic. 2 

Pic.  3 

Pic.l 

Pic.  2 

Pic. 

Face  Group  10 

Lines: 

17 

19 

21 

lU 

20 

- 

13 

13 

- 

12 

12 

20 

Face  Group  11 

Lines : 

17 

19 

21 

11 

11 

3** 
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10 

23 

15 

18 
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Face  Group  12 

Lines: 
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10 
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16 
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17 

15 

Face  Group  13 

Lines : 
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15 

lU 

20 
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- 

Face  Group  14 
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24 
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17 
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5 

- 
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6 

Face  Group  16 
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9 

9 
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6 

6 
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5 

- 

Face  Group  17 

Lines : 
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9 
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5 
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35 

Face  Group  18 
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3 
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7 
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Face  Group  20 
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The  fol loving  face  groups  were  labelled  as  curved:  1,  3,  5.  The  program 
determined  the  following  bodies: 

Body  1 with  face  groups  1 and  2. 

Body  2 with  face  groups  3,  U and  8. 

Body  3 with  face  groups  5,  6 and  7. 

Body  U with  face  groups  9,  10,  11,  12,  13  and  lU. 

Body  5 with  face  groups  15,  l6,  17,  18,  19  and  20. 

6.2  Second  Example 

The  scene  analyzed  for  the  second  example  is  shown  in  Fig.  6.2.  The 
input  data  are  shown  in  Figures  6.3,  6.U , and  6.5.  Part  of  the  data 
recovery  results  is  show,  in  Fig.  6.6.  The  computer  output  is  given  in 
Fig.  6.7  (a)  through  (f). 

6.3  Third  Example 

The  scene  for  the  third  example  is  shown  in  Fig.  6.8.  Input  data  is 
shown  in  Fig's.  6.9,  6.10,  and  6.11.  Part  of  the  data  recovery  results 
is  shown  in  Fig.  6.12.  The  computer  output  is  given  in  Fig.  6.13  (a) 
through  (f).  Of  special  interest  is  the  determination  of  the  curved 
faces. 
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Fig.  6.7  (a)  Computer  Output  Data  for  Example 
(Continued) 


Fig.  6.7  (b)  Computer  Output  Data  for  Example 
(Continued) 
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Fig.  6.7  (f)  Computer  Output  Data  for  Example 
(End) 
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Fig.  6.9  Third  Scene:  Picture-1  Input  Dete 
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Pi«.  6 ,10  Third  Scene:  Picture- 2 Input  Data 
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Fig.  6.11  Third  Scene:  Picture-3  Input  Data 


Fig.  6.13  (a)  Computer  Output  Data  for  Example 
(Continued) 


Fig.  6.13  (c)  Computer  Output  Data  for  Example 
(Continued) 


tIK-  l»  t'lA..'  I’M  iH 


Ul  *-»  — u 

/2  ? / / x. 


<*i  -M  *•  ■*>  f+  w rf>  £ 

<#  *•!—•■#  w <J  ^ u<  OMU 

* g 4 Z € 


2 “ 11 


X o* 


•X  f'* 

<UUbl  t. 

£ ±z* 

1~  x * 


« 


Fig.  6.13  (d)  Computer  Output  Data  for  Example 
(Continued) 
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Fig.  6.13  (*)  Computer  Output  Data  for  Example 
(Continued) 
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VII.  SUMMARY  AND  CONCLUSIONS 


7.1  Summary 

The  objective  of  this  research  was  to  investigate  techniques  for 
the  canputer  understanding  of  pictures  of  scenes.  Three  important  prin- 
ciples guided  the  research:  (a)  No  complete  preknowledge  of  the  scenes' 
bodies  should  be  presumed  since  that  requires  a limited  repertoire  of 
bodies.  Instead,  the  number  of  bodies  should  be  unrestricted  and  merely 
some  general  properties  of  the  bodies  should  be  prespecified,  (b)  These 
properties  should  be  flexible  enough  to  perait  inclusion  of  all  bodies 
likely  to  be  encountered,  (c)  Since  the  input  is  to  be  gotten  from 
photographs,  the  presence  of  data  imperfections  such  as  missing  information, 
wrong  information,  and  geometric  inaccuracies  must  be  taken  into  considera- 
tion. The  difficulty  of  the  described  objective  is  highlighted  by  the 
fact  that  previous  research  in  this  area  always  presupposed  the  placing 
of  severe  restrictions  on  the  problem  environment.  Where  imperfect  data 
was  dealt  with,  only  a small,  fixed  repertoire  of  polyhedra  was  allowed  (6). 
Where  the  fixed  repertoire  was  replaced  by  general  properties  of  a family 
of  bodies  (though  less  general  than  those  assumed  in  this  thesis),  per- 
fect input  data  vas  required  (3,  18). 

To  achieve  the  objective  set  here,  set  of  multiple  pictures  of  the 
scene  vas  assumed  to  serve  as  the  input  data.  New  grammar  rules  needed 
for  the  analysis  of  these  pictures  were,  formalized.  A procedure  for 
establishing  matches  between  features  in  the  different  pictures,  and  for 
verifying  doubtful  features,  were  devised,  as  well  as  procedures  for 
eliminating  wrong  data  and  for  recovering  missing  data.  Finally,  a 
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procedure  for  assembling  the  analyzed  data  into  sets,  each  describing  a 
single  body  of  the  scene,  was  constructed,  including  the  determination 
of  the  bodies'  faces  and  their  nature. 

A computer  program  based  on  this  approach  was  written.  The  pro- 
gram was  able,  given  real  input  data,  to  "understand"  the  photographed 
scenes,  and  to  yield  plausible  descriptions.  The  program  is  cautious  in 
the  decision  making  process,  at  the  expense  of  losing  some  of  the  given 
features.  Thus,  although  the  description  obtained  may  be  incomplete,  no 
errors  are  expected.  A canon  and  crucial  difficulty  encountered  in  the 
preprocessing  stage  is  telling  a T Junction  from  a flat  Y or  W Junction. 
This  can  be  handled  by  the  program,  if  desired,  by  adding  the  flat  Y and 
W Junctions  to  the  Junctions  that  need  verification.  No  restrictions 
need  be  placed  on  the  intersection  of  two  curved  faces;  however,  if  the 
intersection  is  of  a degree  higher  than  that  of  a conic,  the  determina- 
tion of  an  A Junction  may  be  problematic. 

The  main  weakness  of  the  test  of  the  program  is  that  the  program 
was  not  linked  to  an  existing  preprocessor.  It  thus  was  not  subjected 
to  a completed  "reality"  test.  However,  if  and  when  such  a link-up  with 
a preprocessor  is  achieved,  the  power  of  the  main  program  will  grow  in 
the  following  sense:  In  the  stage  of  data  recovery  we  made  a final  Judg- 
. ment  as  to  whether  or  not  a line  should  be  extended  to  a Junction  on 

the  basis  of  the  fitted  equation  and  a set  of  severe  restrictions. 

Once  the  progrsm  is  connected  to  a preprocessor,  this  can  be  replaced 
with  a request  from  the  program  to  the  preprocessor  to  take  a closer 
look  in  the  particular  area  of  Interest.  The  data  recovery  decision  can 
then  be  made  on  the  basis  of  new  more  precisely  determined  data. 
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During  the  course  of  the  development  of  the  matching  procedure,  it 
became  apparent  that  a set  of  three  pictures  is  a good  choice  but  not 
optimal.  If  ve  use  a bigger  set  of  pictures  (covering  approximately  the 
same  visible  range),  ve  are  likely  to  need  less  context  support  and  are 
almost  certain  to  obtain  superior  results. 

7.2  Specific  Contributions 

The  following  are  considered  to  be  the  specific  contributions  pre- 
sented in  this  dissertation. 

1.  New  grammar  rules,  crucial  for  picture  analysis,  for  a family 
of  curved  bodies  were  formulated.  These  rules,  vhen  also 
applied  to  polyhedra,  complement  the  set  of  rules  already  known 
and  strengthen  our  ability  for  machine  analysis  of  pictures 
containing  polyhedral -type  bodies. 

2.  Procedures  were  developed  for  matching  features  in  a set  of 
pictures  taken  from  different  vantage  points,  taking  into  con- 
sideration the  possible  presence  of  serious  errors  in  feature 
extraction. 

3.  A technique  was  developed  for  the  verification  of  doubtful 
features  extracted  from  the  pictures,  and  for  the  elimination 
of  some  possible  preprocessor  errors. 

1*.  Data  correction  and  recovering  techniques  were  developed  having 
the  ability  of  knowledgeably  instructing  the  preprocessor  for 
further  data  extraction  efforts. 


- 103  - 


5.  A procedure  was  formulated  for  assembling  the  analyzed  data 
from  all  pictures  into  sets,  each  representing  a body,  and 
segmenting  into  subsets,  each  representing  a face.  In  addi- 
tion a procedure  was  given  for  determining  the  nature  of  each 
face  (i.e.,  whether  planar  or  curved  and  what  type  of  curve  if 
curved ) . 

6.  A program  was  written  and  tested  for  describing  scenes  of 
curved  bodies  on  the  basis  of  imperfect  data  extracted  from 
multiple  pictures.  Except  for  information  about  some  general 
properties,  the  program  has  no  a priori  knowledge  of  the  bodies. 
The  program  consists  of  about  3,500  PL/I  statements  and  was 
tested  against  data  extracted  from  photographs  of  real  scenes. 

7-3  Directions  for  Future  Research 

In  our  search  to  provide  computers  with  some  kind  of  3D  perception, 
there  is  a strong  tendency  to  imitate  the  way  a human  being's  perception 
abilities  function.  There  are  two  difficulties  with  that  approach. 
Firstly  we  do  not  know  how  a human's  perception  really  works.  All  models 
for  human  perception  presented  thus  far  at  best  are  able  to  explain  only 
a small  part  of  this  ability.  Secondly  it  is  questionable  as  to  whether 
imitating  human  perception  is  the  best  approach  for  developing  machine 
perception.  It  is  the  writer's  feeling  that  a breakthrough  in  machine 
perception,  which  departs  from  the  traditional  approach,  is  needed  and 
due  to  come. 
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Meanwhile,  following  the  present  approach  it  appears  that  consider- 
able improvements  in  performance  can  be  achieved  if  the  preprocessor  is 
incorporated  more  intimately  into  the  overall  process.  It  should  not 
stand  by  itself  but  rather  be  an  integral  part  of  the  whole  picture- 
understanding process.  For  example,  in  the  process  of  determining  an 
intersection  of  lines  to  locate  a junction  (a  painful  task  for  the  pre- 
processor) other  pictures  should  be  consulted,  utilizing  the  geometric 
relations  described  in  the  thesis.  The  process  of  extracting  features 
from  the  pictures  should  not  be  serial  but  rather  parallel,  and 
thoroughly  exploiting  the  matching  tools  described.  Also,  as  already 
mentioned  in  Section  7.1,  the  possibility  should  be  considered  of  using 
pictures  from  more  than  three  vantage  points  to  provide  the  data  for  the 
scene  analysis.  Finally,  thought  should  be  given  to  broaden  and  unify 
the  scene  description  terms.  The  two  present  modes  of  description  - 
faces  and  their  intersections  and  combinations  of  simple  volumes  - 
should  be  used  together,  in  addition  to  other  future  terms,  in  a formal 
scene  description  language. 
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vives outline  information  from  a preprocessor  which  has  extracted  this 
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information  from  the  pictures 


The  outline  information  (set  of  line  structures) 
may  be  imperfect  in  that  some  junctions  may  be  erroneously  reported  and  some 
lines  may  be  missing.  However,  all  lines  due  to  shadows  a£e  assumed  to  have 
been  eliminated  by  the  preprocessor. 


The  theai*  includes  a technique  for  establishing  the  validity  of  the  junctions  pre- 
sented by  the  preprocessor  as  well  as  for  matching  corresponding  features  in 
the  line  structures  derived  from  the  different  pictures.  New  grammar  rules 
for  line-drawing  projections  of  curved  and  planar  solid  bodies  are  developed. 
These  are  useful  in  parsing  the  line  drawings.  They  have  also  led  to  the  defi- 
nition of  a new  family  of  impossible  objects.  The  program  works  simultaneous- 
ly with  all  the  available  line  structures.  The  parsing  of  every  line  structure  is 
supported  dynamically  by  the  results  gotten  thus  far  from  the  parsing  of  the 
other  line  structures.  Through  the  parsing  of  the  line  structures  the  use  of  pic- 
ture comparison  and  the  application  of  the  grammar  rules,  many  of  the  prepro- 
cessor errors  are  detected  and  partly  corrected.  The  program  also  can  pro- 
vide feedback  to  the  preprocessor  in  the  form  of  suggestion  as  to  where  to  look 
again  for  lines  in  the  pictures. 


The  program  utilises  the  extracted  line  structures  corresponding  to  the  differ- 
ent bodies  in  all  the  pictures  to  determine  the  set  of  faces  (insofar  as  possible) 
for  every  body.  Every  face  is  defined  by  an  ordered  set  of  n-tuples.  The  n- 
tuples  are  the  matched  lines  and  junctions  in  the  n different  pictures.  The 
three-dimensional  coordinates  of  the  vertices  and  the  equations  of  the  faces  can 
then  be  determined  from  these  n-tuples.  The  program  was  written  in  PL/I  and 
has  been  tested  on  several  scenes. 
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